setuptool’s “test” command hates atexit

Posted on Posted in Programming

Earlier today, I was writing a Python module that made extensive use of temporary directories.  As you might already know, Python’s ‘tempfile’ module automatically deletes temporary files when it exits, but it does not delete temporary directories.  To work around this quirk, I decided to register an ‘atexit’ event handler to my module.  It basically looked like this:

I ran the library from the Python console several times to verify that it was working correctly.  In every case, the temporary directories were deleted when the program exited.  Awesome!  So, I wrote some unit tests, committed my code, and went on vacation for a week.  Life was good.

But, when I got back from vacation, I noticed something a little strange: there were a LOT of empty temporary directories on the build machine!  Could it be that code rot somehow caused my library to stop working while I was away?  Dismayed with this possibility, I re-ran all of the unit tests, and I retested everything from the Python console.  Phew — everything worked as expected.  I concluded that something else must have been creating the temporary files.

And then I ran the unit tests for a library that used this module.

To my astonishment, none of the temporary directories were deleted!  Could it be that this other library was mucking around with the ‘atexit’ handlers?  Well, no — I had written both of these libraries, so I knew that neither of them were doing anything strange with the ‘atexit’ handlers.  Plus, the problem only occurred when I executed the unit tests in the following manner:

After adding some debugging code, I was able to deduce that Python was not invoking my ‘atexit’ handler.  At this point, I was beginning to suspect that ‘setuptools’ might be registering its own ‘atexit’ handler that called ‘os._exit’.  If this was the case, then any ‘atexit’ handler registered after this one would never be executed.  To test this hypothesis, I added an ‘atexit’ handler to the beginning of the ‘setup.py’ file:

This time, the unit tests produced the following output:

BWA?  Why on earth did my call to ‘log.debug’ fail?  This same logger had been used by several other functions in the code without any problems!  This type of failure would only occur if something systematically went through and deleted every module!

And then I realized that the ‘test’ command in setuptools was systematically going through and deleting every module between test runs.  After digging through the ‘setuptools’ source code for a few minutes, I spotted the following:

Well, that officially confirmed it. Setuptools was mucking with the modules. At first, my ‘atexit’ handlers were failing to execute because the ‘atexit’ module had been unloaded. By adding the ‘import atexit’ statement to the beginning of my ‘setup.py’ file, I had inadvertently made the ‘atexit’ module immune to unloading. Yet, the modules that my ‘atexit’ handler needed were NOT immune.  Therefore, when my ‘atexit’ handler was executed, it attempted to reference objects that no longer existed.  Yikes!

The Solution

Believe it or not, I was able to find a clean way to work around this issue.  It only required 3 steps:

Step 1: Import the ‘atexit’ module inside ‘setup.py’

Placing the following code snippet at the beginning of your ‘setup.py’ file will cause the ‘atexit’ module to become immune to module unloading:

Since the ‘atexit’ module is no longer unloaded, it will invoke all of the ‘atexit’ handlers when the program terminates (yay!). But, setuptools still unloads all of the other modules before this actually occurs. This means that our ‘atexit’ handlers are just going to crash when they are executed (boo!). So…

Step 2: Import the required modules from within the ‘atexit’ handler

By importing the modules we need from within our ‘atexit’ handler, we can overcome most of the grief caused by setuptool’s module unloading.  For example:

Step 3: Refactor the ‘atexit’ handler so that it’s an instance method

But don’t forget — the module that defined our ‘atexit’ handler was also unloaded.  Therefore, any references to attributes defined on this module are going to fail!  We can overcome this problem by changing how these attributes are scoped.  For instance:

Notice that the ‘_temp_folders’ list has been converted from an attribute on a module to a field on a class.  Furthermore, we’re passing ‘atexit.register(…)’ an instance method rather than a module function.  Together, these factors guarantee that ‘self.__temp_folders’ will be in scope when the ‘atexit’ handler is invoked.

Loose Ends

Unfortunately, anyone who uses this library in their own code is going to encounter exactly the same problem.  Why?  Because they’re not going to know that they need to add ‘import atexit’ to the beginning of their ‘setup.py’ file!

It would be nice if setuptools never unloaded the ‘atexit’ module in the first place.  Then again, making this change in setuptools would probably cause many existing unit test suites to begin failing.  Oh well — at least I know my library works correctly when it’s run in production mode 🙂

4 thoughts on “setuptool’s “test” command hates atexit

  1. Sorry for the inconvenience, but this is intended to keep the effects of tests isolated. It never even occurred to me that somebody would put an atexit handler in a test, it seems like a job for tearDown, try/finally, or a with: block.

    That’s probably because it’s really not a good idea to use atexit handlers for cleanup operations in Python in general — as you’ve observed, it’s possible for a callback to be made when its globals are no longer accessible (similar to the case with __del__ methods. Also, you never know how long it’s going to be till the exit actually occurs, or if it will actually be triggered at all! (And not just because of setuptools). I highly recommend not using it if you can avoid it. (And again, not just in the case of setuptools.)

    Setuptools unloads test modules for a number of reasons, one of which is that it allows you to do continuous testing in a single process. You could just watch the source directory and loop, for example. Each time the test command re-runs, it’ll re-import the modules, and thus run on fresh code. (The old tkinter GUI runner for the “unittest” module worked that way – after every time you clicked to run the tests, it reset the state of sys.modules so the next run would be a fresh reload.)

    Of course, that’s not the only reason it’s that way (and at this point the logs would remember better than I what the other reasons are), but as you can imagine, you’d run into exactly this same problem with the tkinter test runner, as it does the same kind of module reset. (Ditto for a lot of code-reloading web servers as well.) So, best not to use atexit:

    1. In tests, or
    2. At all!

    AFAICT, Python’s atexit module is a holdover from a bygone era of C programming which lacked exceptions, try/finally, etc., where it was the most convenient way to handle error cleanup. I’m hard-pressed to remember a single instance in the last 15 years where using it was a better idea than the alternatives.

  2. Hey hey.

    Yeah, I understand why setuptools has this behavior. To clarify: I was not attempting to use setuptools to test the ‘atexit’ handler. One of the libraries I wrote just happened to have an ‘atexit’ handler, and I was surprised to discover that it was not firing when Python was shutting down.

    Unfortunately for me, there is no escaping using the ‘atexit’ method in this particular case. This library was designed to perform a final-pass cleanup of temporary folders in the event that developers forgot to delete them properly (which, unfortunately, tends to be quite frequently). It provided 2 levels of defense:

    • Temporary folders could be created via a context manager. Once the context manager went out of scope, the temporary folders would be deleted automatically
    • If the temporary folders were created outside of the context manager, then the system would keep track of them. If the folders were still present when Python was shutting down, then it would go ahead and remove them.

    This second level of defense relied upon the ‘atexit’ handler. I wrote this library knowing that there would be many cases in which this second level of defense would fail; unfortunately, I did not realize that running the unit tests was going to be one of them.

  3. Thanks for this article. It inspired a rather simple solution. You can manually run all of the atexit functions by calling atexit._run_exitfuncs().

    If you extend the setuptools.command.test command–which is commonplace if you use pytest/nosetest–you can manually call that atexit snippet at the end of test.run_tests before the program exits.

    I’ll paste a sample to a gist: https://gist.github.com/bionikspoon/d3e36133aeba95e46390#file-atexit_setup-py-L15

Leave a Reply

Your email address will not be published. Required fields are marked *