Recently, I made an offhand comment here about:
I’ve actually started thinking about/sketching an actor model build on top of MP, using concepts from actors/monitors and things in the ecosystem today
The ensuing comments and discussion were pretty good - but last night Michael Sparks (of Kamaelia) posted a darned nice comment:
Are you aware that a complete mini-axon using generators is tiny and the rest is optimisations and extra stuff that you find useful in real world systems? By tiny, I mean this small:
* http://www.kamaelia.org/MiniAxonFull
A mini-axon using processes would be equally lightweight (shorter probably) and pretty awesome.
Also, it’s easy to confuse the two halves of Kamaelia. If you think of Kamaelia as just an actor-type implementation, then it’s actually more an actor-like implementation, with STM & an internal SOA system of just over 2000 lines (which is how big Axon actually is, excluding comments & docs), with 80,000 lines of examples…
However, personally I view it as a mechanism for building components which happen to be best used in a concurrent fashion. ie rather than viewing it as “a mechanism for using concurrency”, I view it as “OK, assume we have concurrency, how can we use this to assist in building and maintaining systems”. Axon also gives you the tools for taking these concurrent systems, and interfacing between concurrent systems and standard code. (http://www.kamaelia.org/AxonHandle)
As a result, I view Axon as a library which provides you with the tools wrapping up idioms useful for building collections of components which be a framework.
Anyway, potato/potato, tomato/tomato - if you like, you like, if you don’t, you don’t.
I’d love to replace our existing process based stuff btw with a multiprocessing based version though. If I was going to go down this route, I’d follow our mini axon tutorial to do so, largely becauseit’s essentially the starting point I took with the multiprocess stuff recently and it worked out pretty well.
Beyond this basic stuff though, I’ve noted that people generally start talking about co-ordination languages and building up pattern repositories. The interesting intersection between these two which you get if you call things components rather than actors is it becomes natural to create components called a chassis. These chassis often instantiate directly in concrete usable form concepts that you’d normally refer to as a pattern - Pipeline, Graphline, Carousel, Backplane, Seq, TPipe, etc.
On a random note, you may want to check out MASCOT “Modular Approach to Software Construction, Operation and Test”. I heard about it late last year, and it appears to have the same sort of architecture as Kamaelia. Interestingly (to me) it makes the same key decision - when you send a message outside your component, you don’t know who is going to receive it. This then enables (and requires) a higher level system for connecting components together. The upshot is highly reusable components. This doesn’t entirely surprise me - my ethos came from recognising that asynchronous hardware systems & network systems look strikingly similar… (cf http://www.slideshare.net/kamaelian/sociable-so…)
Anyway, reference for MASCOT: http://async.org.uk/Hugo.Simpson/ - skip down to the end of the page for this PDF: http://async.org.uk/Hugo.Simpson/MASCOT-3.1-Man… I was really pleased to be pointed at MASCOT, largely because it showed a large number of other domains where the same basic model has been used for well over 30 years… Just with non-existent exposure, and slightly different metaphors. Though we, like it, also have mechanisms for automatically visualising systems, with a 1:1 correspondence. Beyond that this also gives us a model that matches Edward Lee’s “The Problem with threads” - we’d released running code long before that paper was published :-)
Anyway, I’m glad that you’re looking at what we’ve done. If you use it, that’d be great, and I’ll happily merge anything you’d like to have a life. (the only comment I’d make there is metaphors and accessibility count - this is surely the point of python? :-) If you don’t take what we use etc but it helps you solidify your thoughts to “No, I don’t want that, I want this”, then likewise, I’m equally glad. If you do that, I’d love to know what you do try, since I like to merge best practice concurrency ideas into Axon :-)
I’d *REALLY* suggest looking at MASCOT though. Really made my Christmas last year when I was pointed at it.
We’re currently having lots of fun using concurrency, primarily by allowing it to make our lives easier, and forgettable about :) It’d be nice to see something similar on top of multiprocessing (which we’ll do if you don’t, but it’d be great if you did - but I’d understand if your view was that you prefer a pure actor (sender knows receiver) model.
Originally posted as a comment by Michael Sparks on jessenoller.com comments using Disqus.
I wanted to pull that comment out and showcase what Michael has to say, and in some way, respond. First, yes - I am looking at Kamaelia (from here on out, I’m going to call it “Kam” - I keep transposing the ae). I actually ran through the mini-axon tutorial, and when I have time, I’m trying to tease apart the internals to better understand it.
Fundamentally, I agree with you (Michael) about the aspects of making concurrency easier (and safer). Right now, I think Kam is a pretty darned good start - for a framework *grin*.
When I made the comment I made, I didn’t think it would get the response it got. There has been a dog-pile of discussions about concurrency best practices/etc and in fact, there’s a discussion still going on on the python-list about concurrency stuff going on right now.
Personally, I only work on the concurrency stuff part-part time - this includes my minor work on Python-Core. My day job is a test engineer - while I am building highly concurrent (and distributed) tests and I use multiprocessing and threading daily, it’s not my full time pursuit. I am passionate about it, and I am passionate about improving python as a language, and library - and if I can do it as a day job and open source it, or get company time to do it, by golly I will.
When I said “I want to build an actor model” - I was not necessarily talking about doing an implementation for python-core. I’m a big believer in learning-through-implementation - so when I said “build x”, not only did I mean “build something for the world” - I also meant “build something for my own benefit” so I can deep-dive into the concepts, problems etc.
This is why I am adverse to jumping in and simply “using” a framework - not because I don’t think it does something exceedingly well - trust me, I am an opportunistic developer - if I can find a library that does what I need *right now* - I’ll use it.
That being said - I am exploring Kamaelia, and yes hopefully I can steal some time to actually do an implementation of the process-based stuff with multiprocessing. I want to explore everything in the ecosystem today - my discussions with Adam Olsen around this stuff (and around python-safethread) and others has made me really want to explore solutions that help everyone, and take the best ideas and concepts and rolls it into something worthy of Python core.
As I have said before - I believe there is room within Python as a language, and CPython as an implementation of that language - for a virtual buffet of helpful libraries for concurrency and distributed systems. Right now, we have threading, async* and multiprocessing. There is plenty of room to grow. Maybe one day I can steal time to grab more of the concepts from java.util.concurrent and propose them via a PEP. Heck - maybe we can work as a group to propose an actor/monitor implementation for Python-Core.
So - personally, and by way of responding to Michael in a more concrete way: I’m, personally looking at anything I can to learn more about implementations and strategies. If something were to come out of it, and I felt strongly enough to propose inclusion in core, I would write and post a PEP - and not run in blindly.
I’ve got more than enough stuff to work on in addition to the day job and being a Dad, and yard work. Oh and the day job, which doesn’t involve nearly as much distributed-and-concurrent systems in Python as I’d like :).
Heck - thinking about it we’d need a good messaging implementation too. I’ll put that on the pile too.
Quick Rant (slightly off topic):
Also, can we stop talking about the damned GIL? Yes, you need locks, No, you probably don’t care about the GIL. Stop yammering about how “broken” CPython is because of it- CPython is an implementation, not the final one and not the only one. If the GIL really gets you excited, either drop to a C module, use multiprocessing or something else. The GIL is here to stay for some time - either propose a PEP (and a Patch) that doesn’t break CPython or hush. Enough bike-shedding - discussion is great, especially when something comes out of it, but constantly berating/lamenting things is just a bike shed. The shed is purple, now move on. Purple!
Required Reading ( in addition to Michael’s links):
Another information rich issue of 
