jessenoller.com

Doug’s fol­lowed up last week’s OS mod­ule post with another tid­bit around pipe cre­ation.

Per­son­ally, I would avoid os.popen* com­pletely, and go with the lat­est sub­process mod­ule for pipe con­trol. The sub­process mod­ule cleans up a lot of the rough edges around pipe/subprocess cre­ation found in the os.popen and popen2 modules.

See Doug’s post reminded me about some­thing I wanted to blog about a lit­tle while ago — cer­tain ver­sions (up to 2.4.4) of sub­process and popen2.* are not thread­safe. See python bug 1183780 — “Popen4 wait() fails spo­rad­i­cally with threads”. This chewed us up for awhile at work — we do a lot of heav­ily threaded popen2 spawn­ing⁴⁴ and we were see­ing this error:

File "/usr/lib/python2.4/popen2.py", line 94, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

At an alarm­ing rate. I pulled down the lat­est python sub­ver­sion tree and tracked down the change list/version of popen2 that had the fix for us and with some site.addsitedir() magic⁴⁴we now replace the 2.4.x bro­ken ver­sion with our own patched ver­sion that works.

This is some­thing a lot of peo­ple should watch out for — espe­cially linux users that are on older ver­sions of 2.4. If you think you are hit­ting this — just sync the python sub­ver­sion, grab the updated ver­sion for your python ver­sion and either over­write the one the stdlib, or make sure it’s sym­bol loads last in sys.path via sitecustomize.py or sys.path.insert(0, ‘yourpathhere’)

Edit to add: Just so every­one can see an exam­ple — this exper­i­ment was done using Fedora Core 6’s default python instal­la­tion — I can down­load the exam­ple script and exac­er­bate the prob­lem like this:

[jesse@lol ~]# python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[jesse@lol ~]# python popen_bug.py -n 100
Exception in thread Thread-11:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "popen_bug.py", line 43, in run
pipe.wait()
File "/usr/lib/python2.4/popen2.py", line 94, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

I’ve tested the patched ver­sion up to 1000 threads.
4

1. we haven’t had time to switch to sub­process — ah, legacy code444
2. by the way, the site mod­ules doc­u­men­ta­tion in the library ref­er­ence sucks.444

4

Doug’s fol­lowed up last week’s OS mod­ule post with another tid­bit around pipe cre­ation. Per­son­ally, I would avoid os.popen* com­pletely, and go with the lat­est sub­process mod­ule for pipe con­trol....