Skip to content

A followup on Concurrency within Python

Coming out of PyCon 2008 my head is again filled with thoughts on concurrency based on the talks and work I was involved with at the Con.

First up: I want to point out something which bothered me a little. Namely, that many people seemed resigned and pushing towards Jython/IronPython as the “concurrency answer”. There was a lot of discussion/buzz about the fact that both of these implementations are the future of concurrency for python as a language due to the obvious lack of a GIL.

In discussing this with many people, I may have come off as defensive if nothing more than for the fact I constantly refuted that as “the answer”. Don’t get me wrong, I am eager, no, very eager to get my hands on the latest version of Jython to reach in and try out the java.util.concurrent package via Jython, I think access to that library will be great, for those working in a hybrid Java/Python shop.

But I don’t think “Python’s” future should be tightly coupled with the implementation of a runtime on top of another runtime. I think work has to be performed on the CPython interpreter to make it a viable contender and solution within the “concurrency” space. Python’s strength will be found in multiple strong implementations of the interpreter.

Let me be clear: I believe Python programmers want to write Python – not Java, nor .Net. What attracts us to Python is a clean syntax. If I want concurrency in Python, I don’t want to have to call into java.util.concurrent. Obviously, the Jython guys are not going to force this – threading will simply use Java threading, but I think a “pythonic” abstraction of the java.util.concurrent package as a whole would also be desired – so you could really use the power of the libraries, but stay wrapped in the warm pastry of Python.

Concurrency and distributed design is a large enough space that the use-cases and problems people want and need to address open the playing field for not just “one” solution (i.e: getting rid of the GIL) but a series of language and standard library improvements to Python as a whole.

My lightning talk touched on this: I think there are existing projects that are well-suited for inclusion into the standard library, and I believe Adam Olsen’s work on the safe threading project (update here) will also help pave the way for a much more appealing future for CPython – even if it is in Py3k.

Python has a history of picking the best ideas from other languages and improving on them. For example: Python probably does not want to simply re-create the java.util.concurrent package – the concurrent package views threads as the solution. Python would be well suited to support “real threads” in as simple and an abstracted way as possible (a leaky abstraction at that).

Erlang has the Actor/Asynchonus Message passing model (a functional language implementation) (for more on Erlang concurrency, see this). While people may be envious of what they see as “perfect side effect free concurrency” – I doubt they envy the syntax or complex nature of it. Still yet – there are existing fork+exec+message passing libraries available for python right now that sidestep the GIL, but allow you to keep basic threading primitives and semantics and parallelize to cores, and clusters.

In discussion all of this with people at PyCon, I really began to realize that many people in the community look in envy to other language’s concurrency implementations – including Java’s Threads (even though many people still insist threads are impossible to get right).

I also realized that a lot of people are trying to solve a lot of various problems. When people hear distributed, concurrency and parallelism, they immediately bring up large data crunching (ala map reduce and its kin), distributed filesystems (ala GFS and Hadoop) to simple examples of jobs spread across multiple cores for math crunching and jobs passed to entire clusters of hundreds of machines.

Everyone is looking for an answer to their problem – and Python can not implement something that will address your problem. It can only provide the platform with the tools to allow you to solve it yourself.

I suggested the following to multiple people at pycon – this is what I view as the “cleanest” approach to providing the abstractions and tools people will need to move forward with CPython in this space.

  1. Adopt/move forward with a version of CPython based on the work/concepts offered in Adam Olsen’s Safe/Without GIL Threading work. Adopt the monitor and deadlock work for all of CPython, but leave the GIL removal as a compile-time option (python3.0 and python3.0-mt binaries).
  2. Add the pyProcessing, pprocess, or Parallel-Python module to the standard library for those who want to use vanilla cpython, and to whom the fork method works. Which module is added should use existing Python semantics, but be powerful enough for advance/distributed usage. Note, I have started work on a PEP to get the pyProcessing module into the stdlib.
  3. Add an Actor – or Actor-Like module to the functools module, in keeping with the concepts of the current implementations. Python’s implementation would be obviously leaky. (Note, I am not a functional programmer by any stretch, I like my OOP, suggestions welcome). This could easily take advantage of both the Safe Threading work.
  4. Add a lightweight (not XML) messaging system to couple with the Actor implementation . (no, I have no suggestions here)

Obviously, a varied approach like this doesn’t conflict with the “one way to do it right” methodology of Python, in a problem space as varied as this – “one way to do it right” is completely dependent on the problem being solved. It can also help couch the discussion of what the problem they’re trying to solve really is.

I’ve obviously been trying to help with the safe threading work – and I am writing a PEP for the inclusion of pyProcessing into the stdlib, but even this work is not the end – it’s simply a start to help give people some batteries to plug into their concurrent toy car.

We should think about the building blocks to help people solve whatever problem that arises – we can’t add a solution that fits any one problem, we can only provide the tools with which to build a great big concurrent future.

Now I have to go – my daughter has acquired the lock.

Also, thanks to Adam for helping review this!

  • MichaelSparks
    Hi there,

    You may find Kamaelia interesting - I've been adding clean multicore support recently. You can find an overview of its design philosophy on our summer of code page here: http://kamaelia.sourceforge.net/SummerOfCode2008 (linking there because there's an embedded presentation & lots of links).

    You can find discussion of how multicore support was added here:
    http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=...
    Rather than pyprocessing it uses pprocess since that seems sufficient. I'll take a look at pyprocessing though.

    Single process pipeline:
    Pipeline(
    Textbox(position=(20, 340),
    text_height=36,
    screen_width=900,
    screen_height=400,
    background_color=(130,0,70),
    text_color=(255,255,255)),
    TextDisplayer(position=(20, 90),
    text_height=36,
    screen_width=400,
    screen_height=540,
    background_color=(130,0,70),
    text_color=(255,255,255))
    )

    Multiprocess pipeline:
    ProcessPipeline(
    Textbox(position=(20, 340),
    text_height=36,
    screen_width=900,
    screen_height=400,
    background_color=(130,0,70),
    text_color=(255,255,255)),
    TextDisplayer(position=(20, 90),
    text_height=36,
    screen_width=400,
    screen_height=540,
    background_color=(130,0,70),
    text_color=(255,255,255))
    )


    It just works. It seems to have similar inspiration as erlang, but I wasn't aware of erlangs execution model when I created Kamaelia. (It was more inspired by asynchronous hardware systems, unix pipelines, CSP and biological systems).

    cf http://kamaelia.sourceforge.net/Introduction

    I went into that (due to work modelling biological systems) with the assumption that whilst the primary communication would be akin to the nervous system (send and receive of pulses of information between processors) hence why the core of kamaelia is called Axon, that there would need to be an equivalent for a high latency, low use, but useful hormonal system. That boils down to a global key value store. (which will be migrating soon to an STM based store).

    The interesting thing from my perspective is that this also mirrors a system from RAF Malvern from 30 years ago, called MASCOT which I heard about at christmas. That uses channels, instead of named inboxes/outboxes, and pools which are equivalent to our CAT ("hormonal" systems). MASCOT is utterly fascinating because they started from the same premise. I only recently heard about MASCOT (just before christmas), and the only reference I can find online is this, but if you scroll down, you'll find "The Official Handbook of Mascot : Version 3.1 : June 1987". Perhaps scarily, their first version appears to have been in 1975....)

    http://async.org.uk/Hugo.Simpson/

    OK, I'll get back to suggesting ideas for students for GSOC which are naturally highly concurrent, fun, in python beginner friendly and naturally multicore.

    Incidentally, there is also another difference between python & concurrency in a functional language - the fact that objects in python are mutable. I blogged about some implications of this here:
    http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=...

    Kamaelia has proof of concept implementations in Java, C++ and ruby as well.

    Have fun :-)

    Michael.
  • Kamaelia *does* look interesting. I am going to have to check it out.
  • Thank you for speaking sense about concurrency in Python. The project hosted at the website listed with this post would benefit in a large way from a CPython without a GIL. I am glad someone else shares the view that python should give you tools to build your application, your way, not dictate its design.

    The code for the robotics system above uses more C++ code then needed because it needs true concurrency. A python without a GIL would let us coders benefit from the reduction in code size you get when you go from C++ to python.

    -Joseph Lisee
  • The biggest thing to remember about the GIL is that is it not a boogeyman, but rather an implementation detail of the interpreter. It's design is such that it keeps the C implementation simple especially for extension writers.

    Removing the GIL, if done poorly, will grossly increase the complexity of the interpreter, slow C development down and make extension writing a nightmare.

    It's important to realize the benefits it does provide.
  • qwerty
    Although pyprocessing isn't the end-all solution, it's syntax is really easy to pick up. I really hope you manage to get it into the standard library.
  • So do I - and I agree, it's not the final solution, it's only a library to help people get "a" (not The) job done.
  • masklinn
    > I doubt they envy the syntax or complex nature of it

    What syntax wouldn't there be to envy, and which part of Erlang's concurrency model is complex?

    First point: Erlang's concurrency requires 2 primitives: a "send message to process" and a "retrieve message from mailbox". The first one is straightforward and trivial, the second one is a bit more complex (it needs the ability to have timeouts, and while pattern matching makes filtering/fetching messages extremely straightforward it may be more complex to handle that without PM). The send/receive syntax is one of the best parts of erlang's syntax: straightforward, short and to the point. What isn't there to envy? Yes we could add `spawn`, but that's not specific to erlang-style concurrency.

    Second point: the complexity. In all of my experiments, Erlang-style concurrency was infinitely simpler than e.g. java-style concurrency, even with java.util.concurrent (1.5's, we haven't switched to 1.6 yet) for various reasons:

    * A single concurrency structure (which can be built upon to e.g. supervisor tree, but at its core Erlang only has "send a message, receive a message")

    * Unconstrained resources, there is no need to bother with thread-pooling and other tripe/implementation details in Erlang. If you need a process, just spawn it and let the runtime take care of its allocation and scheduling

    * Synchronous/sequential erlang and asynchronous/concurrent erlang are clearly separated with function calls on one side and messages on the other one, message sends don't look like function calls and reciprocally.

    * Finally, erlang has the tools to handle massive concurrency, in order to introspect and debug a system that can have thousands of running processes.

    So I'd like to know more about what you consider to be the syntactical or complexity problems of shared-nothing message-passing concurrency *compared to shared-memory "java-style"* concurrency.
  • It's not the implementation of Erlang that I'm critiquing - I'm very much pro-shared-little/nothing, ergo my liking of Adam Olsen's monitor work in the safethreading project. I have a bone to pick with erlang's syntax itself.

    I agree with you on the power of Erlang's model - however I think the barrier-to-entry of Erlang is why it continues to largely be a niche language. If Python were to adopt some of the theory behind what Erlang is done, I would be happy.
blog comments powered by Disqus