Multiprocessing in hindsight.

by jesse in ,

Steve Holden shot an email out to python dev last night. In it, he said something which at first tweaked me, but in thinking about it, I both see, and some part of me agrees with his point. To quote:

I think that both 3.0 and 2.6 were rushed releases. 2.6 showed it in the inclusion (later recognizable as somewhat ill-advised so late in the day) of multiprocessing; 3.0 shows it in the very fact that this discussion has become necessary. So we face an important turning point: is 3.1 going to be serious production quality or not?

I sent a reply out to python-list just a few minutes ago (actually, two of them) - the first is this, which I'll quote verbatim:

I might write a longer blog post about this later, but I can see Steve's point of view. The fact is, pyprocessing/multiprocessing was a late addition to Python 2.6. Personally, I was game to put it into either 2.7 or 2.6, but I felt inclusion into 2.6 wasn't completely out of question - and others agreed with me.

See these mail threads:

And so on.

All of that being said; the initial conversion and merging of the code into core exposed a lot of bugs I and others didn't realize were there in the first place. I take full responsibility for that - however some of those bugs were in python-core itself (deadlock after fork anyone?).

So, the road to inclusion was a bit rougher than I initially thought - I relied heavily on the skills of people who had more experience in the core than I did, and it was disruptive to the release schedule of python 2.6 due to both the bugs and instability.

I however; disagree that this was ultimately a bad decision, or that it was some how indicative of a poorly managed or rushed 2.6 release. All releases have bugs, and towards the end of the 2.6 cycle, multiprocessing *was not* the release blocker.

After 2.6 went out, I had a small wave of bugs filed against multiprocessing that I've been working through bit by bit (I still need to work on BSD/Solaris issues) and some of the bugs have exposed issues I simply wish weren't there but I think this is true of any package, especially one as complex as multiprocessing is.

I know of plenty of people using the package now, and I know of several groups switching to 2.6 as quickly as possible due to its new features, bug fixes/etc. Multiprocessing as a package is not bug free - I'm the first to admit that - however it is useful, and being used and frankly, I maintain that it is just one step in a larger project to bring additional concurrency and distributed "stuff" into python-core over time.

So yes, I see Steve's point - multiprocessing *was* disruptive, and it inclusion late in the game siphoned off resources that could have been used elsewhere. Again, I'll take the responsibility for soiling the pool this way. I do however think, that python 2.6 is overall a *fantastic* release both feature wise, quality wise and is quite useful for people who want to "get things done" (tm).

Now I'm going to go back to fixing bugs.


And the second one (which is a reply to a followup by steve) is here.

I thought I might expound on this just a little bit - the inclusion of multiprocessing was my first real exposure to anything other than being a read-only descriptor to python-core. In my excitement to get multiprocessing in, having used it quite a bit and discussed with other people it's benefits, I pushed putting it into 2.6 pretty strongly.

The fact is, that taking a module from 3rd-partyness, and migrating it into the python-core ecosystem, especially something with multiple platform nuances/a lot of C code and API cleanup/reworking/etc, is a big task.

If we had done a straight port from pyprocessing -> multiprocessing, it would have been simpler, but we took the chance to alter the APIs to be more pythonic and PEP 8 compliant, as well as refactoring to take advantage of python 2.6/3.0-isms, rewrote a good chunk of the docs/etc.

Doing this much work late in a release cycle is not cool, especially when I (the guy on the hook) is working on it sub-part time, and relying heavily on people in python-core for help.

The amount of help from people like Benjamin, Antoine, Nick, Amaury, and Mark D., Adam, was simply awesome, and really pulled my ass out of the fire - nothing quite like a noob to disrupt things, eh?

In hindsight (oh how I wish I had you sooner), targeting 2.7 would probably have been better, but I and others didn't want to delay inclusion for +1 year given the fact that concurrency is such a hot-button issue.

Did I underestimate the amount of work? Yes. Were there/are there bugs? Yes. Do I continue to rely on guidance and help from people with bigger brains than I have? Yes.

Additionally, I should point out: I did not write pyprocessing. Richard Oudkerk did, and he did an excellent job in doing so. But something people may not realize, is that shortly after he and I went down this path, and we started doing initial code drops into core, Richard ran into connectivity problems and, well - fell off the internets. I haven't heard from him in some time.

So, in all of this, I lost the one person who knew the code base best of all - but we trucked through, and as I said before, with the help of others, we got it in, we got it stable, and we got 2.6 out.

I continue to fix bugs, and I continue to rely heavily on community-provided patches (this is my part-part time gig after all) - especially when it comes to operating systems which *cough* are "interesting". I have a wishlist of enhancements to make, and a pile of things I still need to learn.

Over time, I would like to factor-out some of the more "magical" things the module does, and completely rewrite the test suite. Did you know that multiprocessing alone has 124 different test? I'd like to double that by the end of the year. I'd also like to revamp the documentation, which isn't as newbie-friendly as it should be.

From a test engineer standpoint: I did the one thing I know in my heart should never be done - push a saint bernard of a feature through the cat door of a release. You should never increase your risk by adopting a large feature late in a date driven release. Ever. Doing so drastically increases risk, instability and siphons off resources you need elsewhere. I rant and rail about this at jobs all the time; yet I ignored it in this case. Shame on me.

In closing; can you use it? Yes. I know of plenty of people who do. Are there bugs? Yes. If you find them, file them. I am proud of where it is today, but I am also not naive enough to think it's done.