Pycon: Unladen-Swallow

March 26th, 2009 § 21 comments


Tree-Swallow.jpgSo, by now some par­tic­u­lar set of peo­ple (mainly those at the VM-Summit and twit­ter) have heard about unladen-swallow, a new project out of “the Google” which is work­ing on pro­vid­ing some seri­ous speed increases to the CPython interpreter.

This is being worked on by Collin Win­ter, Jef­fery Yasskin and Thomas Wouters — it’s a branch of CPython: Not a Fork. Some of the improve­ments could pos­si­bly be rapidly inte­grated to python-trunk, some of them (such as using LLVM) are a longer road obvi­ously, but given the peo­ple involved, and oth­ers in that arena, I could eas­ily see this sup­plant­ing the cur­rent inter­preter quickly.

But I’m biased, because they sped up CPickle (which is what mul­ti­pro­cess­ing uses for shar­ing data between processes). Oh, and they include psyco (port to 64 bit ok please).

The goals are nice, quot­ing a few choice ones from the project plan:

We want to make Python faster, but we also want to make it easy for large, well-established appli­ca­tions to switch to Unladen Swallow.

  1. Pro­duce a ver­sion of Python at least 5x faster than CPython.
  2. Python appli­ca­tion per­for­mance should be stable.
  3. Main­tain source-level com­pat­i­bil­ity with CPython applications.
  4. Main­tain source-level com­pat­i­bil­ity with CPython exten­sion modules.
  5. We do not want to main­tain a Python imple­men­ta­tion for­ever; we view our work as a branch, not a fork.

And (from 2009 Q3 Goals):

In addi­tion, we intend to remove the GIL and fix the state of mul­ti­thread­ing in Python. We believe this is pos­si­ble through the imple­men­ta­tion of a more sophis­ti­cated GC sys­tem, some­thing like IBM’s Recy­cler (Bacon et al, 2001).

Our long-term goal is to make Python fast enough to start mov­ing performance-important types and func­tions from C back to Python.

The great thing is: They have a work­ing imple­men­ta­tion right now. Yessir, it’s not vapor! Hooray!

I’ve got it down, com­piled and I’m futz­ing around with it now, yes it works. Unfor­tu­nately test­ing it I found a bug in mul­ti­pro­cess­ing (not unladen). Damn!

  • Pingback: Michael Tsai - Blog - Unladen Swallow

  • Paul Bod­die

    Inter­est­ing fig­ures on the “Releases” page, although I’m not sure about their speed-up terminology:

    1.023 -> 0.409: 150.36% faster

    Surely, if the fig­ures on the left are times, the speed-up is around two-fold in this case. Per­son­ally, I dis­like the com­bi­na­tion of per­cent­ages with superla­tives — one should say “40% of the orig­i­nal time” or “2.x of the orig­i­nal speed”.

    Any­way, let’s wait for it to get into (someone’s edi­tion of) the shootout. ;-)

  • http://jessenoller.com jnoller

    Drop an email to their list and ask for a clarification

  • Mar­tin

    Wow. Seems very pro­fes­sional. Will be inter­est­ing to fol­low the development.

  • llim­l­lib

    They say on that page that the times are cal­cu­lated as (new-old)/new, so (.409 — 1.023) / .409 = –1.5 -> 150%

    I agree that this is silly; .409 is 2.5 times faster than 1.023, not 1.5 times.

  • http://inhahe.blogspot.com inhahe

    that’s pretty cool. PyPy is already tar­get­ing LLVM, afaik, i don’t know why they didn’t just donate a lot of work to PyPy.

    also i’m not sure why mul­ti­thread­ing is so impor­tant. the GIL applies to threads, but not to processes. just use the the pp mod­ule or sim­i­lar. i haven’t done much par­al­lel pro­gram­ming but they always scream in #python never to use thread­ing because any kind of scal­ing intro­duces really hard-to-track bugs. i think they favor IPC over shared memory.

    for a python imple­men­ta­tion the main thing i’d be con­cerned with is mostly elim­i­nat­ing hash table lookups for attrib­utes. i won­der if their project does that.

  • aono

    100% faster == 2x speedup. 150% faster == speedup of 2.5

    I agree that ‘speedup’ is the cor­rect met­ric, but a lot of peo­ple dont get it, and x% faster should con­vey what they want. Its just not imme­di­ately obvi­ous with 150%, though it would with 100%

  • http://jessenoller.com jnoller

    PyPy is a lot more, and a lot less than what they’re try­ing to do. PyPy is a com­plete re-implementation of “the world” — unladen swal­low is per­for­mance enhance­ments. Also, AFAIK, they dropped the LLVM work.

    Also, Hi, I’m the mul­ti­pro­cess­ing mod­ule main­tainer (par­al­lel pro­gram­ming with processes in python 2.6), too. As for #python hat­ing threads: not every­one has a ratio­nal view on threads vs. processes vs every­thing else. There’s a place in the world for both.

  • http://www.ironpythoninaction.com/ Michael Foord

    The cool thing is that Unladen swal­low has been serv­ing YouTube traf­fic since last Friday!

  • http://docwhat.gerf.org docwhat

    So how does this relate to Parrot?

  • http://www.ironpythoninaction.com/ Michael Foord

    It doesn’t really relate to Par­rot. This is a branch of CPython. The imple­men­ta­tion of Python on Par­rot is called Pynie but there is a long way to go with it.

  • http://docwhat.gerf.org docwhat

    They are sim­i­lar in that both are tar­get­ing new vir­tual machines, are they not? I don’t know how sim­i­lar they are (or aren’t). I know both are reg­is­tered based.

  • Luis

    Please tell us more! Where did you get that??

  • Luis

    @Michael: Where did you get that? Could you tell us more about this being used in Youtube??

  • neu­russ

    Please tell us more !! Where did you het that??

  • Pingback: procoders.net » Python 5 times faster ?

  • http://autiomaa.org Daniel Schildt

    Sounds inter­est­ing. Hope­fully they can get things work­ing nicely and in as sta­ble way as pos­si­ble. Hope­fully this can increase amount of inter­est peo­ple have in learn­ing of Python.

  • http://www.voidspace.org.uk/python/weblog/index.shtml Michael Foord

    From the google guys who announced unladen-swallow at the Python VM sum­mit at PyCon.

  • http://www.voidspace.org.uk/python/weblog/index.shtml Michael Foord

    Par­rot *is* a Vir­tual machine — for run­ning mul­ti­ple lan­guages. The Python port for Par­rot (Pynie) is very young. unladen-swallow is an attempt (amongst other things) to inte­grate parts of the LLVM into core CPython to pro­vide a speedup. They are unre­lated projects with dif­fer­ent goals — but there is a VM involved in both…

  • http://www.voidspace.org.uk/python/weblog/index.shtml Michael Foord

    From the google guys who announced unladen-swallow at the Python VM sum­mit at PyCon.

  • http://www.voidspace.org.uk/python/weblog/index.shtml Michael Foord

    Par­rot *is* a Vir­tual machine — for run­ning mul­ti­ple lan­guages. The Python port for Par­rot (Pynie) is very young. unladen-swallow is an attempt (amongst other things) to inte­grate parts of the LLVM into core CPython to pro­vide a speedup. They are unre­lated projects with dif­fer­ent goals — but there is a VM involved in both…

What's this?

You are currently reading Pycon: Unladen-Swallow at jessenoller.com.

meta