jessenoller.com

Orig­i­nally, this post was going to be much more dif­fer­ent than what is has become — the orig­i­nal title was “Fail­ing in Pub­lic” — but I don’t think “fail­ing” is fair to me per­son­ally, or to any­one who has ever helped me, or con­tributed a patch or a fix to the mul­ti­pro­cess­ing module.

Yes­ter­day, I made a state­ment on twit­ter:

I am offi­cially look­ing for some­one to take over mul­ti­pro­cess­ing main­te­nance from me. http://bugs.python.org/issue6721

Ignor­ing any com­ments in that bug; I main­tain that a later tweet is still true:

Some­times good points and poignant crit­i­cism can be buried in a pile of crap.

In hind­sight; I could have worded the orig­i­nal mes­sage dif­fer­ently “tak­ing over main­te­nance” means that I am, and always have been the sole con­trib­u­tor to the mul­ti­pro­cess­ing code base, which is patently false. Antoine, and many other python core devel­op­ers, and peo­ple within the com­mu­nity have sub­mit­ted bug reports, patches, tests and doc­u­men­ta­tion. My words were inten­tion­ally harsh — but the direc­tion of that harsh­ness was to me; I feel that as the “leader” (for some mea­sure­ment of “lead”) I have been remiss in my respon­si­bil­i­ties and leadership.

Sure; I could be less harsh on myself — but the level of expec­ta­tional debt that I’ve incurred against myself for the mod­ule and the main­te­nance has grown, and grown. Even if I find myself lead­ing PyCon, busy as a PSF Direc­tor, push­ing the core-mentors pro­gram, the sprints pro­gram, and a lot of other com­mu­nity projects, I am still respon­si­ble for the care and feed­ing for the crea­ture I helped cre­ate and birth. I’ve com­mit­ted the sin of “going dark”.

For some his­tory, see:

• Mul­ti­pro­cess­ing in Hindsight
• Python 3.0, some mul­ti­pro­cess­ing info, admin­is­tra­tive notes.
• The 2.6/3.0 Mul­ti­pro­cess­ing todo list
• Pro­gram­mer Inse­cu­rity and Mea Culpa

Months ago — I spun up the multiprocessing-sig mail­ing list, in hopes to engage more peo­ple — highly active users, inter­ested peo­ple, etc to help me pay down the debt. Of course, in ret­ro­spect; it’s unfair for me to expect any­one but me to help me pay down the debt I’ve incurred. On the other hand, the respon­si­ble thing for me to do — the mature thing for me to do — is to ask for help — not to “wash my hands” of any­thing, but rather to take this as an oppor­tu­nity to look as mul­ti­pro­cess­ing as some­thing greater than what I orig­i­nally envi­sioned and sub­mit­ted to core.

I hear from peo­ple every day who are using the mod­ule — every day, some­thing I helped birth helps peo­ple get things done. Mul­ti­pro­cess­ing has grown up by virtue of becom­ing part of Python core, and daily — despite the bugs, the debt and the quirks — it helps devel­op­ers achieve some­thing they might have oth­er­wise been unable (or at least, had a more dif­fi­cult time) to do.

The mod­ule is expan­sive — it has pools, tools for dis­trib­uted pro­gram­ming via man­agers, pipes for inter­process com­mu­ni­ca­tions, it’s fea­ture set is both large, and ulti­mately com­plex in its under­pin­nings. That com­plex­ity — that fea­ture set — is the rea­son why that debt, the bugs, the quirks has grown over time. If it wasn’t being used — I wouldn’t have so many emails about it — or bugs filed against it.

So where are we/it today?

Today, mul­ti­pro­cess­ing has wide­spread usage — in Python 3, there’s actu­ally a new mod­ule named concurrent.futures that builds on the build­ing blocks of mul­ti­pro­cess­ing and thread­ing. Pack­ages like Cel­ery use it exten­sively (and work around inter­nal quirks). For Python 3 — the sky is the future for what mul­ti­pro­cess­ing could be — addi­tional func­tion­al­ity, mov­ing parts of it (such as the pool abstrac­tions) into the con­cur­rent name­space, extend­ing and improv­ing the Man­ager classes, etc. For Python 2.7 — bug fixes, doc fixes only.

If you search the Python bug tracker for the word “mul­ti­pro­cess­ing” regard­less of assignee, you’ll get 119 hits. That’s right; 119 — not all of them are mul­ti­pro­cess­ing bugs — and many of them are dupes, or fixed in recent ver­sions. What that query gets you is an idea of the debt that has to be paid down and resolved. Each one of those bugs needs to be looked at, repro­duced, de-duped and patched. Some of them may be doc­u­men­ta­tion issues, some are pretty hairy (like the afore­men­tioned http://bugs.python.org/issue6721 as well as http://bugs.python.org/issue4106 and http://bugs.python.org/issue8713).

What I ask­ing for — rather than wash­ing my hands of any­thing, or any attempt to absolve myself of respon­si­bil­ity, is for help. I am stretched thin — too thin to do this myself, or to be the only per­son who can main­tain, under­stand or work on this mod­ule. It’s too big for that, it’s too impor­tant for me to be the arbiter of it any longer. It’s big­ger than me.

Aside from the bug queue; there’s a short list of things that need to be done — the docs need to have a fresh, hard set of eyes on them, there are things (behav­iors, fea­tures) that are undoc­u­mented. The test suite needs a com­plete over­haul — when I inher­ited the code, this is the first thing I should have done — but I didn’t. The prob­lem is that the test suite is mired in magic and com­plex­ity, and with­out an expan­sive, main­tain­able test suite, I don’t feel con­fi­dent that the bug list can be addressed with confidence.

So, I come to you with my hat in my hands, a hum­bled man. It’s unrea­son­able for me to ask for oth­ers to “pay down” the debt I’ve incurred; but it’s irre­spon­si­ble, imma­ture and mis­guided of me to think that I alone, or any sin­gle per­son can go at this alone. So I need your help — and, if in time, some­one choses to be the “leader” for the mod­ule, then I will gladly step back. Until then, I will try to con­tinue to be a guid­ing hand and at least point peo­ple in the proper direc­tion, com­mit patches, etc.

If you are inter­ested; please speak up — or just wade into the bug queue. You can sign up to the multiprocessing-sig list, and ask ques­tions there, or if you’re new to core Python, and want some addi­tional men­tor­ship, check out the Python Core Men­tor­ship pro­gram — that list serves as a gen­tle and polite, wel­com­ing intro­duc­tion to core devel­op­ment. No ques­tion — no mat­ter how green — is off lim­its, and it’s already got an excel­lent track record of help­ing peo­ple get up to speed.

In clos­ing; I’m going to apol­o­gize — we all know lives change, careers change, and inter­ests change. All of these things have hap­pened to me, but in chang­ing so quickly and tak­ing on dif­fer­ent roles, I left some­thing impor­tant behind. In doing so, I have done a great dis­ser­vice to you, the com­mu­nity and users.

I will also thank you; with­out you — the users, cur­rent and future helpers, mul­ti­pro­cess­ing wouldn’t exist or be rel­e­vant in any con­text. With­out you, I wouldn’t have the drive to even write this post, fight to get mul­ti­pro­cess­ing into the std lib to begin with, or per­form any of the other roles I do.

So; thank you.

Orig­i­nally, this post was going to be much more dif­fer­ent than what is has become — the orig­i­nal title was “Fail­ing in Pub­lic” — but I don’t think “fail­ing” is...