jessenoller.com

Note:This is another post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more./Note

Stack­less python — here’s another big one on the pile — is much more than a library, or a frame­work which runs on CPython — Stack­less is actu­ally a mod­i­fied ver­sion of the CPython inter­preter. It’s much more than just a C-extension. Stack­less is in use by var­i­ous peo­ple and com­pa­nies — most notably, it’s in use by CCP Games, mak­ers of Eve Online (see this pycon pre­sen­ta­tion). In fact, CCP Games is a large part of why Stack­less is still around today.

» Read the rest of this entry «

Note:This is another post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the...

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on python-list, but more recently, I know he’s been work­ing on get­ting some level of mul­ti­pro­cess­ing into cir­cuits — cir­cuits was already on my research list, but I bumped it up on the queue because we started chatting.

Let me state this: these are slightly cleaned up ver­sions of my notes as I am learn­ing these mod­ules — some of them have higher learn­ing curves for me than others.

Cir­cuits is, well — an event based “frame­work” (again, note the small f) based around the con­cept of Com­po­nents (big C!) consuming/reacting and in turn gen­er­at­ing events — all asynchronously.

James’ goals seem pretty sim­ple — build some­thing with no exter­nal depen­den­cies, that’s com­pact (I should say, the core.py is < 500 lines) and that makes it easy to build scal­able mes­sag­ing based (event based) sys­tems. Did he suc­ceed — don’t know! Let’s dig in.

» Read the rest of this entry «

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on...

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more. Code and exam­ples will be checked into my pycon 2009 bit­bucket site here/Note

For awhile now, I’ve been mean­ing to dig into Kamaelia but was largely put off by what I tend to call the “twisted effect”. What this means is that when I go look­ing for libraries and small com­po­nents, I go look­ing for a library — not a “solu­tion”. I also worry about the “once you go in, you must fol­low this par­a­digm” effect. I’m not going to say that these feel­ing con­tinue to be founded, or are com­pletely ratio­nal — after all, I am dig­ging into it, yes? It’s the thought that once you adopt the “one true way of doing things” you’re trapped in that solution/framework “for­ever” — iron­i­cally, I love Django for it’s “con­cep­tual integrity” and full-stack approach. No, I don’t under­stand me either.

» Read the rest of this entry «

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing...

Recently, I made an off­hand com­ment here about:

I’ve actu­ally started think­ing about/sketching an actor model build on top of MP, using con­cepts from actors/monitors and things in the ecosys­tem today

The ensu­ing com­ments and dis­cus­sion were pretty good — but last night Michael Sparks (of Kamaelia) posted a darned nice comment:

Are you aware that a com­plete mini-axon using gen­er­a­tors is tiny and the rest is opti­mi­sa­tions and extra stuff that you find use­ful in real world sys­tems? By tiny, I mean this small:
* http://www.kamaelia.org/MiniAxonFull

A mini-axon using processes would be equally light­weight (shorter prob­a­bly) and pretty awesome.

Also, it’s easy to con­fuse the two halves of Kamaelia. If you think of Kamaelia as just an actor-type imple­men­ta­tion, then it’s actu­ally more an actor-like imple­men­ta­tion, with STM & an inter­nal SOA sys­tem of just over 2000 lines (which is how big Axon actu­ally is, exclud­ing com­ments & docs), with 80,000 lines of examples…

How­ever, per­son­ally I view it as a mech­a­nism for build­ing com­po­nents which hap­pen to be best used in a con­cur­rent fash­ion. ie rather than view­ing it as “a mech­a­nism for using con­cur­rency”, I view it as “OK, assume we have con­cur­rency, how can we use this to assist in build­ing and main­tain­ing sys­tems”. Axon also gives you the tools for tak­ing these con­cur­rent sys­tems, and inter­fac­ing between con­cur­rent sys­tems and stan­dard code. (http://www.kamaelia.org/AxonHandle)

As a result, I view Axon as a library which pro­vides you with the tools wrap­ping up idioms use­ful for build­ing col­lec­tions of com­po­nents which be a framework.

Any­way, potato/potato, tomato/tomato — if you like, you like, if you don’t, you don’t.

I’d love to replace our exist­ing process based stuff btw with a mul­ti­pro­cess­ing based ver­sion though. If I was going to go down this route, I’d fol­low our mini axon tuto­r­ial to do so, largely becauseit’s essen­tially the start­ing point I took with the mul­ti­process stuff recently and it worked out pretty well.

Beyond this basic stuff though, I’ve noted that peo­ple gen­er­ally start talk­ing about co-ordination lan­guages and build­ing up pat­tern repos­i­to­ries. The inter­est­ing inter­sec­tion between these two which you get if you call things com­po­nents rather than actors is it becomes nat­ural to cre­ate com­po­nents called a chas­sis. These chas­sis often instan­ti­ate directly in con­crete usable form con­cepts that you’d nor­mally refer to as a pat­tern — Pipeline, Graphline, Carousel, Back­plane, Seq, TPipe, etc.

On a ran­dom note, you may want to check out MASCOT “Mod­u­lar Approach to Soft­ware Con­struc­tion, Oper­a­tion and Test”. I heard about it late last year, and it appears to have the same sort of archi­tec­ture as Kamaelia. Inter­est­ingly (to me) it makes the same key deci­sion — when you send a mes­sage out­side your com­po­nent, you don’t know who is going to receive it. This then enables (and requires) a higher level sys­tem for con­nect­ing com­po­nents together. The upshot is highly reusable com­po­nents. This doesn’t entirely sur­prise me — my ethos came from recog­nis­ing that asyn­chro­nous hard­ware sys­tems & net­work sys­tems look strik­ingly sim­i­lar… (cf http://www.slideshare.net/kamaelian/sociable-so…)

Any­way, ref­er­ence for MASCOT: http://async.org.uk/Hugo.Simpson/ — skip down to the end of the page for this PDF: http://async.org.uk/Hugo.Simpson/MASCOT-3.1-Man… I was really pleased to be pointed at MASCOT, largely because it showed a large num­ber of other domains where the same basic model has been used for well over 30 years… Just with non-existent expo­sure, and slightly dif­fer­ent metaphors. Though we, like it, also have mech­a­nisms for auto­mat­i­cally visu­al­is­ing sys­tems, with a 1:1 cor­re­spon­dence. Beyond that this also gives us a model that matches Edward Lee’s “The Prob­lem with threads” — we’d released run­ning code long before that paper was published :-)

Any­way, I’m glad that you’re look­ing at what we’ve done. If you use it, that’d be great, and I’ll hap­pily merge any­thing you’d like to have a life. (the only com­ment I’d make there is metaphors and acces­si­bil­ity count — this is surely the point of python? :-) If you don’t take what we use etc but it helps you solid­ify your thoughts to “No, I don’t want that, I want this”, then like­wise, I’m equally glad. If you do that, I’d love to know what you do try, since I like to merge best prac­tice con­cur­rency ideas into Axon :-)

I’d *REALLY* sug­gest look­ing at MASCOT though. Really made my Christ­mas last year when I was pointed at it.

We’re cur­rently hav­ing lots of fun using con­cur­rency, pri­mar­ily by allow­ing it to make our lives eas­ier, and for­get­table about :) It’d be nice to see some­thing sim­i­lar on top of mul­ti­pro­cess­ing (which we’ll do if you don’t, but it’d be great if you did — but I’d under­stand if your view was that you pre­fer a pure actor (sender knows receiver) model.

Orig­i­nally posted as a com­ment by Michael Sparks on jessenoller.com com­ments using Dis­qus.

I wanted to pull that com­ment out and show­case what Michael has to say, and in some way, respond. First, yes — I am look­ing at Kamaelia (from here on out, I’m going to call it “Kam” — I keep trans­pos­ing the ae). I actu­ally ran through the mini-axon tuto­r­ial, and when I have time, I’m try­ing to tease apart the inter­nals to bet­ter under­stand it.

Fun­da­men­tally, I agree with you (Michael) about the aspects of mak­ing con­cur­rency eas­ier (and safer). Right now, I think Kam is a pretty darned good start — for a frame­work *grin*.

When I made the com­ment I made, I didn’t think it would get the response it got. There has been a dog-pile of dis­cus­sions about con­cur­rency best practices/etc and in fact, there’s a dis­cus­sion still going on on the python-list about con­cur­rency stuff going on right now.

Per­son­ally, I only work on the con­cur­rency stuff part-part time — this includes my minor work on Python-Core. My day job is a test engi­neer — while I am build­ing highly con­cur­rent (and dis­trib­uted) tests and I use mul­ti­pro­cess­ing and thread­ing daily, it’s not my full time pur­suit. I am pas­sion­ate about it, and I am pas­sion­ate about improv­ing python as a lan­guage, and library — and if I can do it as a day job and open source it, or get com­pany time to do it, by golly I will.

When I said “I want to build an actor model” — I was not nec­es­sar­ily talk­ing about doing an imple­men­ta­tion for python-core. I’m a big believer in learning-through-implementation — so when I said “build x”, not only did I mean “build some­thing for the world” — I also meant “build some­thing for my own ben­e­fit” so I can deep-dive into the con­cepts, prob­lems etc.

This is why I am adverse to jump­ing in and sim­ply “using” a frame­work — not because I don’t think it does some­thing exceed­ingly well — trust me, I am an oppor­tunis­tic devel­oper — if I can find a library that does what I need *right now* — I’ll use it.

That being said — I am explor­ing Kamaelia, and yes hope­fully I can steal some time to actu­ally do an imple­men­ta­tion of the process-based stuff with mul­ti­pro­cess­ing. I want to explore every­thing in the ecosys­tem today — my dis­cus­sions with Adam Olsen around this stuff (and around python-safethread) and oth­ers has made me really want to explore solu­tions that help every­one, and take the best ideas and con­cepts and rolls it into some­thing wor­thy of Python core.

As I have said before — I believe there is room within Python as a lan­guage, and CPython as an imple­men­ta­tion of that lan­guage — for a vir­tual buf­fet of help­ful libraries for con­cur­rency and dis­trib­uted sys­tems. Right now, we have thread­ing, async* and mul­ti­pro­cess­ing. There is plenty of room to grow. Maybe one day I can steal time to grab more of the con­cepts from java.util.concurrent and pro­pose them via a PEP. Heck — maybe we can work as a group to pro­pose an actor/monitor imple­men­ta­tion for Python-Core.

So — per­son­ally, and by way of respond­ing to Michael in a more con­crete way: I’m, per­son­ally look­ing at any­thing I can to learn more about imple­men­ta­tions and strate­gies. If some­thing were to come out of it, and I felt strongly enough to pro­pose inclu­sion in core, I would write and post a PEP — and not run in blindly.

I’ve got more than enough stuff to work on in addi­tion to the day job and being a Dad, and yard work. Oh and the day job, which doesn’t involve nearly as much distributed-and-concurrent sys­tems in Python as I’d like :).

Heck — think­ing about it we’d need a good mes­sag­ing imple­men­ta­tion too. I’ll put that on the pile too.

Quick Rant (slightly off topic):

Also, can we stop talk­ing about the damned GIL? Yes, you need locks, No, you prob­a­bly don’t care about the GIL. Stop yam­mer­ing about how “bro­ken” CPython is because of it– CPython is an imple­men­ta­tion, not the final one and not the only one. If the GIL really gets you excited, either drop to a C mod­ule, use mul­ti­pro­cess­ing or some­thing else. The GIL is here to stay for some time — either pro­pose a PEP (and a Patch) that doesn’t break CPython or hush. Enough bike-shedding — dis­cus­sion is great, espe­cially when some­thing comes out of it, but con­stantly berating/lamenting things is just a bike shed. The shed is pur­ple, now move on. Purple!

Required Read­ing ( in addi­tion to Michael’s links):

• Kamaelia
• python-safethread
• Twisted
• Drama­tis
• Par­ley
• Mes­sage Pass­ing Con­c­cur­rency (Actor Model) in Python
• Toward Bet­ter Con­cur­rency
• The Pil­lars of Con­cur­rency

Recently, I made an off­hand com­ment here about: I’ve actu­ally started think­ing about/sketching an actor model build on top of MP, using con­cepts from actors/monitors and things in the ecosys­tem today...