jessenoller.com

Hooray, Actors!

A coworker just dropped an ancient artifact off for me to read:

Actors, concurrency and Kamaelia

Recently, I made an offhand comment here about:

I've actually started thinking about/sketching an actor model build on top of MP, using concepts from actors/monitors and things in the ecosystem today

The ensuing comments and discussion were pretty good - but last night Michael Sparks (of Kamaelia) posted a darned nice comment:

Are you aware that a complete mini-axon using generators is tiny and the rest is optimisations and extra stuff that you find useful in real world systems? By tiny, I mean this small:
* http://www.kamaelia.org/MiniAxonFull

A mini-axon using processes would be equally lightweight (shorter probably) and pretty awesome.

Also, it's easy to confuse the two halves of Kamaelia. If you think of Kamaelia as just an actor-type implementation, then it's actually more an actor-like implementation, with STM & an internal SOA system of just over 2000 lines (which is how big Axon actually is, excluding comments & docs), with 80,000 lines of examples...

However, personally I view it as a mechanism for building components which happen to be best used in a concurrent fashion. ie rather than viewing it as "a mechanism for using concurrency", I view it as "OK, assume we have concurrency, how can we use this to assist in building and maintaining systems". Axon also gives you the tools for taking these concurrent systems, and interfacing between concurrent systems and standard code. (http://www.kamaelia.org/AxonHandle)

As a result, I view Axon as a library which provides you with the tools wrapping up idioms useful for building collections of components which be a framework.

Anyway, potato/potato, tomato/tomato - if you like, you like, if you don't, you don't.

I'd love to replace our existing process based stuff btw with a multiprocessing based version though. If I was going to go down this route, I'd follow our mini axon tutorial to do so, largely becauseit's essentially the starting point I took with the multiprocess stuff recently and it worked out pretty well.

Beyond this basic stuff though, I've noted that people generally start talking about co-ordination languages and building up pattern repositories. The interesting intersection between these two which you get if you call things components rather than actors is it becomes natural to create components called a chassis. These chassis often instantiate directly in concrete usable form concepts that you'd normally refer to as a pattern - Pipeline, Graphline, Carousel, Backplane, Seq, TPipe, etc.

On a random note, you may want to check out MASCOT "Modular Approach to Software Construction, Operation and Test". I heard about it late last year, and it appears to have the same sort of architecture as Kamaelia. Interestingly (to me) it makes the same key decision - when you send a message outside your component, you don't know who is going to receive it. This then enables (and requires) a higher level system for connecting components together. The upshot is highly reusable components. This doesn't entirely surprise me - my ethos came from recognising that asynchronous hardware systems & network systems look strikingly similar... (cf http://www.slideshare.net/kamaelian/sociable-so...)

Anyway, reference for MASCOT: http://async.org.uk/Hugo.Simpson/ - skip down to the end of the page for this PDF: http://async.org.uk/Hugo.Simpson/MASCOT-3.1-Man... I was really pleased to be pointed at MASCOT, largely because it showed a large number of other domains where the same basic model has been used for well over 30 years... Just with non-existent exposure, and slightly different metaphors. Though we, like it, also have mechanisms for automatically visualising systems, with a 1:1 correspondence. Beyond that this also gives us a model that matches Edward Lee's "The Problem with threads" - we'd released running code long before that paper was published :-)

Anyway, I'm glad that you're looking at what we've done. If you use it, that'd be great, and I'll happily merge anything you'd like to have a life. (the only comment I'd make there is metaphors and accessibility count - this is surely the point of python? :-) If you don't take what we use etc but it helps you solidify your thoughts to "No, I don't want that, I want this", then likewise, I'm equally glad. If you do that, I'd love to know what you do try, since I like to merge best practice concurrency ideas into Axon :-)

I'd *REALLY* suggest looking at MASCOT though. Really made my Christmas last year when I was pointed at it.

We're currently having lots of fun using concurrency, primarily by allowing it to make our lives easier, and forgettable about :) It'd be nice to see something similar on top of multiprocessing (which we'll do if you don't, but it'd be great if you did - but I'd understand if your view was that you prefer a pure actor (sender knows receiver) model.

Originally posted as a comment by Michael Sparks on jessenoller.com comments using Disqus.

I wanted to pull that comment out and showcase what Michael has to say, and in some way, respond. First, yes - I am looking at Kamaelia (from here on out, I'm going to call it "Kam" - I keep transposing the ae). I actually ran through the mini-axon tutorial, and when I have time, I'm trying to tease apart the internals to better understand it.

Fundamentally, I agree with you (Michael) about the aspects of making concurrency easier (and safer). Right now, I think Kam is a pretty darned good start - for a framework *grin*.

When I made the comment I made, I didn't think it would get the response it got. There has been a dog-pile of discussions about concurrency best practices/etc and in fact, there's a discussion still going on on the python-list about concurrency stuff going on right now.

Personally, I only work on the concurrency stuff part-part time - this includes my minor work on Python-Core. My day job is a test engineer - while I am building highly concurrent (and distributed) tests and I use multiprocessing and threading daily, it's not my full time pursuit. I am passionate about it, and I am passionate about improving python as a language, and library - and if I can do it as a day job and open source it, or get company time to do it, by golly I will.

When I said "I want to build an actor model" - I was not necessarily talking about doing an implementation for python-core. I'm a big believer in learning-through-implementation - so when I said "build x", not only did I mean "build something for the world" - I also meant "build something for my own benefit" so I can deep-dive into the concepts, problems etc.

This is why I am adverse to jumping in and simply "using" a framework - not because I don't think it does something exceedingly well - trust me, I am an opportunistic developer - if I can find a library that does what I need *right now* - I'll use it.

That being said - I am exploring Kamaelia, and yes hopefully I can steal some time to actually do an implementation of the process-based stuff with multiprocessing. I want to explore everything in the ecosystem today - my discussions with Adam Olsen around this stuff (and around python-safethread) and others has made me really want to explore solutions that help everyone, and take the best ideas and concepts and rolls it into something worthy of Python core.

As I have said before - I believe there is room within Python as a language, and CPython as an implementation of that language - for a virtual buffet of helpful libraries for concurrency and distributed systems. Right now, we have threading, async* and multiprocessing. There is plenty of room to grow. Maybe one day I can steal time to grab more of the concepts from java.util.concurrent and propose them via a PEP. Heck - maybe we can work as a group to propose an actor/monitor implementation for Python-Core.

So - personally, and by way of responding to Michael in a more concrete way: I'm, personally looking at anything I can to learn more about implementations and strategies. If something were to come out of it, and I felt strongly enough to propose inclusion in core, I would write and post a PEP - and not run in blindly.

I've got more than enough stuff to work on in addition to the day job and being a Dad, and yard work. Oh and the day job, which doesn't involve nearly as much distributed-and-concurrent systems in Python as I'd like :).

Heck - thinking about it we'd need a good messaging implementation too. I'll put that on the pile too.

Quick Rant (slightly off topic):

Also, can we stop talking about the damned GIL? Yes, you need locks, No, you probably don't care about the GIL. Stop yammering about how "broken" CPython is because of it- CPython is an implementation, not the final one and not the only one. If the GIL really gets you excited, either drop to a C module, use multiprocessing or something else. The GIL is here to stay for some time - either propose a PEP (and a Patch) that doesn't break CPython or hush. Enough bike-shedding - discussion is great, especially when something comes out of it, but constantly berating/lamenting things is just a bike shed. The shed is purple, now move on. Purple!

Required Reading ( in addition to Michael's links):

• Kamaelia
• python-safethread
• Twisted
• Dramatis
• Parley
• Message Passing Conccurrency (Actor Model) in Python
• Toward Better Concurrency
• The Pillars of Concurrency

Guido answers your questions…

Via GVR's blog he's put answers up on the ask-a-google-engineer page for the top 20 questions.

I'm seriously laughing here:

• Q: "Why google choose python as the main programming?"
• A: Incorrect this is.

Oh, and yay multiprocessing (see: this question) but man is there a lot of work to do. I've actually started thinking about/sketching an actor model build on top of MP, using concepts from actors/monitors and things in the ecosystem today

October Issue of Python Magazine is live.

Another information rich issue of Python Magazine is live! This time, it's the October (stunning, I know) issue. The cover story is on "Versioning your database with sqlalchemy-migrate" - and of course, there's a short article from me on "SSH Programming with Paramiko". You can see the run down for the article here.

A lot of good people put a lot of work into the magazine - Doug Hellmann doesn't let any of us slack off, and keeps us on target. If you don't already have a subscription - you should get one.

Additionally, if you like writing - or even if you're new to writing, if you have and idea for an article please submit it!

Also remember, the people behind Python Magazine are having a conference in november - PyWorks is happening in Atlanta, GA on November 12-14th. I'll be doing a talk there on threading/multiprocessing.

Updated pyjavaproperties

I just pushed a minor update for PyJavaProperties out - this just adds a simple list as an index of the keys as they're parsed, and appends new keys on the internal index. Next, I want to keep original comments and whitespace.

Nose-testconfig version .5 uploaded.

Fixes a minor issue with python config file parsing.

Next up, hierarchical YAML files!

Backport of 2.6 Multiprocessing to 2.4/2.5

Thanks to the work of Christian Heimes and Skip Montanaro with a supporting role by me, the 2.6 version of Multiprocessing (pyprocessing) has been forked/adjusted for compatibility with Python 2.4/2.5.

You can see the pypi package here, and the google code project here.

We chose to do the back port for a variety of reasons - having the new API for the package makes it easier to jump into 2.6 if you're like me: a heavy pyprocessing user, not to mention that during the migration into python-core, a lot of bugs were fixed.

Check it out, and feel free to file bugs. We're planning on back porting bug fixes from python-core to the project, and applicable bug fixes from the google code project into python-core.

A nose plugin to run JUnit tests via Jython.

Floating an idea to all of you in pythonland - I've been sketching out a nose plugin that would be able to find and execute JUnit tests in your for your java code base.

The reporting of the result would be unified - using nose-xunit for non-junit tests, you should get back one happy-shiny report for all of your tests.

Anyone have any thoughts out there?

New project: pyjavaproperties

I just uploaded a new pypi package for "pyjavaproperties" - this is a fork of this recipe with the authors permission.

I'm working on a small number of improvements, but my real reason to do this was that - 1: I have to work with properties files. 2: I need it up there so it can be easy_installed and 3: It makes sense as a project!

In any case, here's the pypi page: http://pypi.python.org/pypi/pyjavaproperties/0.1

And the google code page: http://code.google.com/p/pyjavaproperties/

Suggestions welcome. I am going to monkey in support for this in nose-testconfig as soon as I have some spare cycles.

Hooray, retirement!

I don't talk politics or current events here - mainly because I'd come off a a Lolbertarian whack-job, but I just had a serious "what in the crap?! moment when I checked my 401k:

Personal Rate of Return from 01/01/2008 to 10/09/2008 is -41.9%

This is one of the moments I look at my daughter and think about the future and get really sullen, so instead, I'm going to go back to solving technical problems as I can't even rationally discuss this in my own head.