jessenoller.com

Doug’s followed up last week’s OS module post with another tidbit around pipe creation.

Personally, I would avoid os.popen* completely, and go with the latest subprocess module for pipe control. The subprocess module cleans up a lot of the rough edges around pipe/subprocess creation found in the os.popen and popen2 modules.

See Doug’s post reminded me about something I wanted to blog about a little while ago - certain versions (up to 2.4.4) of subprocess and popen2.* are not threadsafe. See python bug 1183780 - “Popen4 wait() fails sporadically with threads”. This chewed us up for awhile at work - we do a lot of heavily threaded popen2 spawning³³ and we were seeing this error:

File "/usr/lib/python2.4/popen2.py", line 94, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

At an alarming rate. I pulled down the latest python subversion tree and tracked down the change list/version of popen2 that had the fix for us and with some site.addsitedir() magic³³we now replace the 2.4.x broken version with our own patched version that works.

This is something a lot of people should watch out for - especially linux users that are on older versions of 2.4. If you think you are hitting this - just sync the python subversion, grab the updated version for your python version and either overwrite the one the stdlib, or make sure it’s symbol loads last in sys.path via sitecustomize.py or sys.path.insert(0, ‘yourpathhere’)

Edit to add: Just so everyone can see an example - this experiment was done using Fedora Core 6’s default python installation - I can download the example script and exacerbate the problem like this:

[jesse@lol ~]# python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>>
[jesse@lol ~]# python popen_bug.py -n 100
Exception in thread Thread-11:
Traceback (most recent call last):
File “/usr/lib/python2.4/threading.py”, line 442, in __bootstrap
self.run()
File “popen_bug.py”, line 43, in run
pipe.wait()
File “/usr/lib/python2.4/popen2.py”, line 94, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

I’ve tested the patched version up to 1000 threads.
3

1. we haven’t had time to switch to subprocess - ah, legacy code333
2. by the way, the site modules documentation in the library reference sucks.333

3