Python 3.0, some multiprocessing info, administrative notes.

December 4th, 2008 § 7 comments

So, first off — unless you’ve lived under a rock for the last 24 hours, you should know Python 3000 final is hot off the bit presses. This marks a huge mile­stone for the lan­guage, and major props are deserved to all of the python-core peo­ple who have spent so much time work­ing on it.

Python 3000 marks an inter­est­ing point in the evo­lu­tion of the lan­guage — we all know it is meant to clean up some of the warts of the python 2.x series. It’s designed and imple­mented know­ing full well it breaks back­wards com­pat­i­bil­ity — but I would argue it doesn’t fall in the black and white camp of rev­o­lu­tion vs. evo­lu­tion. In my mind, Python 3 actu­ally falls right in the mid­dle. Yes, it is rev­o­lu­tion­ary in the aspect that it breaks com­pat­i­bil­ity with 2, but it is not so sig­nif­i­cant a series of break­ages that it falls out­side of the evo­lu­tion­ary camp.

For exam­ple, changes to fun­da­men­tals — like white­space, drop­ping tuples, switch­ing to pure func­tional pro­gram­ming — that in my mind counts as rev­o­lu­tion­ary. Python 3 counts as more of an evo­lu­tion­ary jump — the changes are meant to clean things up — remov­ing the pre­hen­sile tail and sin­gle eye­brow, so to speak.

It’s still a jump though — that why the core team has been repeat­ing the mantra “use it for porting/prototyping but not day to day pro­duc­tion use”. I can’t stress this enough — Python 3 is slower that 2.x, and library main­tain­ers are going to lag sig­nif­i­cantly behind. Sure — down­load it, exper­i­ment with it, and keep it in your mind, but don’t deploy with it — right now.

Actual wide­spread adop­tion of it is going to hap­pen after 3.1/3.2 at very least, and the 2.6 and so on releases will con­tinue to thrive. Adop­tion of 3 will be a long time in com­ing, and will ulti­mately be dri­ven by the OS main­tain­ers. I can eas­ily see, over the next few years, python2 and python3 co-existing on oper­at­ing sys­tems and then one day, python2 just goes away.

All in all, it’s a great series of changes, and a lot of peo­ple put a lot of work and time into it. Python devel­op­ment — core devel­op­ment is prag­matic, rea­soned and always thought­ful about the changes that go into the lan­guage and libraries. The amount of time that is required to be this way is amazing.

Mov­ing on.

So, I spent a few days in novem­ber at PyWorks in Atlanta — I got to do my talk on threading/multiprocessing (slides here). I think it went OK, but pro­vided my mul­ti­pro­cess­ing talk is accepted for PyCon, I know what I will do differently.

I got gen­er­ally pos­i­tive feed­back on the talk, so that was a bonus.

Also, recently I got a chance to fix a hand­ful of mul­ti­pro­cess­ing doc issues — I expanded some exam­ples, fixed some other doc errors/etc. Doc changes go live quickly, so you can see them shortly after I update them.

I am always look­ing for:

  • Bug reports on multiprocessing
  • Usabil­ity issues
  • Exam­ples that would help make things eas­ier to understand
  • Doc suggestions

Every­thing is fair game. I openly admit that not every­thing is as clear as it should be, and the pack­age has some rough edges I want to whit­tle down, but find­ing ran­dom state­ments on pop­u­lar sites like:

except that the mul­ti­pro­cess­ing library is half-baked. it’s been the bane of my exis­tence recently.”

and not hav­ing bug reports, emails or any other infor­ma­tion to help me make it bet­ter dri­ves me up a wall.

Since 2.6 was released, I haven’t had enough time to give it a lot of the TLC it needs — real work, and being a dad trump my open source time. Even my other projects are in the cooler until a less busy time is achieved.

So please — send me you sug­ges­tions, send me your crit­i­cisms, hell, send me hate mail if you really feel the need to vent, but don’t do noth­ing. Right now, I’m the pri­mary main­tainer, but Chris­t­ian Heimes has also been doing a lot of heavy lift­ing, as well as oth­ers. It’s really a team effort (espe­cially as Richard, the orig­i­nal author has been MIA for some time).

Lastly, I fixed my host­ing issues — appar­ently my host dis­abled my account with­out telling me that it was being dis­abled. Hooray.

  • jnoller

    Good idea, gen­er­ally speak­ing though, it’s in python core — the python
    mail­ing list(s) (python dev, python-list) and the bug­tracker are the places
    to go. We might be able to add one to the back­port list, but that mud­dies
    the water.

  • jnoller

    I don’t use iPython: So I don’t know “what it needs for ipython” — some­one
    who uses ipython should do that work. As for lambda sup­port: I don’t get
    what you’re say­ing. How does mul­ti­pro­cess­ing pre­clude use of lambdas?

  • Robert Kern

    Some­thing that would help a lot with get­ting feed­back about mul­ti­pro­cess­ing is a mail­ing list. Even just a note on the project page say­ing “Email feed­back to Jesse Noller at <…>” would work, but a pub­lic, archived mail­ing list is ideal.

  • http://gael-varoquaux.info Gael Varo­quaux

    Talk­ing about mak­ing mul­ti­pro­cess­ing bet­ter, please, please, pretty please with sugar on top, make it work in ipython. Sci­en­tific users are very much inter­ested in mul­ti­pro­cess­ing, but they really rely on ipython for inter­ac­tive work.
    Another nice thing would be sup­port for lamb­das (I think IPython1 has achieved this). The num­ber one use­case is to use map on func­tions with sev­eral arguments.

  • Mike Hansen

    Hi Gael,

    Sage has been using mul­ti­pro­cess­ing from within in IPython for a num­ber of months now. The key thing is that we use some cus­tom code for pick­ling func­tions. Usu­ally it’s used as a dec­o­ra­tor, but you can use it like the following:

    sage: f = lambda x: x*x
    sage: pf = parallel(2)(f) #two processes
    sage: list(pf(range(10)))
    [(((0,), {}), 0),
    (((1,), {}), 1),
    (((2,), {}), 4),
    (((3,), {}), 9),
    (((4,), {}), 16),
    (((5,), {}), 25),
    (((6,), {}), 36),
    (((7,), {}), 49),
    (((8,), {}), 64),
    (((9,), {}), 81)]

    I’m assum­ing that you’d want it so that you don’t have to have this cus­tom pick­ling code in there?

    –Mike

  • Har­ald Armin Massa

    Jesse,

    do you know of any­one who has ported

    http://www.cherokee-project.com/download/pyscgi/

    pyscgi to use pro­cess­ing / multiprocessing?

    SCGI on Win­dows cur­rently fails to use processes as of miss­ing pickle sup­port for sock­ets, and mul­ti­pro­cess­ing / pro­cess­ing explic­itly con­tains them …

    In other words:
    to have a “Mul­ti­pro­cess­ing­Mixin” within SocketServer.py in stan­dard library would be a GREAT idea :)

    Best wishes

    Har­ald

  • http://nutritionfoods09.blogspot.com/ nutri­tion foods

    Thanks for the info. May God have mercy on us all.

What's this?

You are currently reading Python 3.0, some multiprocessing info, administrative notes. at jessenoller.com.

meta