Skip to content

Python 3.0, some multiprocessing info, administrative notes.

So, first off - unless you’ve lived under a rock for the last 24 hours, you should know Python 3000 final is hot off the bit presses. This marks a huge milestone for the language, and major props are deserved to all of the python-core people who have spent so much time working on it.

Python 3000 marks an interesting point in the evolution of the language - we all know it is meant to clean up some of the warts of the python 2.x series. It’s designed and implemented knowing full well it breaks backwards compatibility - but I would argue it doesn’t fall in the black and white camp of revolution vs. evolution. In my mind, Python 3 actually falls right in the middle. Yes, it is revolutionary in the aspect that it breaks compatibility with 2, but it is not so significant a series of breakages that it falls outside of the evolutionary camp.

For example, changes to fundamentals - like whitespace, dropping tuples, switching to pure functional programming - that in my mind counts as revolutionary. Python 3 counts as more of an evolutionary jump - the changes are meant to clean things up - removing the prehensile tail and single eyebrow, so to speak.

It’s still a jump though - that why the core team has been repeating the mantra “use it for porting/prototyping but not day to day production use”. I can’t stress this enough - Python 3 is slower that 2.x, and library maintainers are going to lag significantly behind. Sure - download it, experiment with it, and keep it in your mind, but don’t deploy with it - right now.

Actual widespread adoption of it is going to happen after 3.1/3.2 at very least, and the 2.6 and so on releases will continue to thrive. Adoption of 3 will be a long time in coming, and will ultimately be driven by the OS maintainers. I can easily see, over the next few years, python2 and python3 co-existing on operating systems and then one day, python2 just goes away.

All in all, it’s a great series of changes, and a lot of people put a lot of work and time into it. Python development - core development is pragmatic, reasoned and always thoughtful about the changes that go into the language and libraries. The amount of time that is required to be this way is amazing.

Moving on.

So, I spent a few days in november at PyWorks in Atlanta - I got to do my talk on threading/multiprocessing (slides here). I think it went OK, but provided my multiprocessing talk is accepted for PyCon, I know what I will do differently.

I got generally positive feedback on the talk, so that was a bonus.

Also, recently I got a chance to fix a handful of multiprocessing doc issues - I expanded some examples, fixed some other doc errors/etc. Doc changes go live quickly, so you can see them shortly after I update them.

I am always looking for:

  • Bug reports on multiprocessing
  • Usability issues
  • Examples that would help make things easier to understand
  • Doc suggestions

Everything is fair game. I openly admit that not everything is as clear as it should be, and the package has some rough edges I want to whittle down, but finding random statements on popular sites like:

“except that the multiprocessing library is half-baked. it’s been the bane of my existence recently.”

and not having bug reports, emails or any other information to help me make it better drives me up a wall.

Since 2.6 was released, I haven’t had enough time to give it a lot of the TLC it needs - real work, and being a dad trump my open source time. Even my other projects are in the cooler until a less busy time is achieved.

So please - send me you suggestions, send me your criticisms, hell, send me hate mail if you really feel the need to vent, but don’t do nothing. Right now, I’m the primary maintainer, but Christian Heimes has also been doing a lot of heavy lifting, as well as others. It’s really a team effort (especially as Richard, the original author has been MIA for some time).

Lastly, I fixed my hosting issues - apparently my host disabled my account without telling me that it was being disabled. Hooray.

  • Robert Kern
    Something that would help a lot with getting feedback about multiprocessing is a mailing list. Even just a note on the project page saying "Email feedback to Jesse Noller at <...>" would work, but a public, archived mailing list is ideal.
  • Good idea, generally speaking though, it's in python core - the python
    mailing list(s) (python dev, python-list) and the bugtracker are the places
    to go. We might be able to add one to the backport list, but that muddies
    the water.
  • Talking about making multiprocessing better, please, please, pretty please with sugar on top, make it work in ipython. Scientific users are very much interested in multiprocessing, but they really rely on ipython for interactive work.
    Another nice thing would be support for lambdas (I think IPython1 has achieved this). The number one usecase is to use map on functions with several arguments.
  • I don't use iPython: So I don't know "what it needs for ipython" - someone
    who uses ipython should do that work. As for lambda support: I don't get
    what you're saying. How does multiprocessing preclude use of lambdas?
  • Mike Hansen
    Hi Gael,

    Sage has been using multiprocessing from within in IPython for a number of months now. The key thing is that we use some custom code for pickling functions. Usually it's used as a decorator, but you can use it like the following:

    sage: f = lambda x: x*x
    sage: pf = parallel(2)(f) #two processes
    sage: list(pf(range(10)))
    [(((0,), {}), 0),
    (((1,), {}), 1),
    (((2,), {}), 4),
    (((3,), {}), 9),
    (((4,), {}), 16),
    (((5,), {}), 25),
    (((6,), {}), 36),
    (((7,), {}), 49),
    (((8,), {}), 64),
    (((9,), {}), 81)]

    I'm assuming that you'd want it so that you don't have to have this custom pickling code in there?

    --Mike
  • Harald Armin Massa
    Jesse,

    do you know of anyone who has ported

    http://www.cherokee-project.com/download/pyscgi/

    pyscgi to use processing / multiprocessing?

    SCGI on Windows currently fails to use processes as of missing pickle support for sockets, and multiprocessing / processing explicitly contains them ...

    In other words:
    to have a "MultiprocessingMixin" within SocketServer.py in standard library would be a GREAT idea :)

    Best wishes

    Harald
blog comments powered by Disqus