jessenoller.com

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more. Code and exam­ples will be checked into my pycon 2009 bit­bucket site here/Note

For awhile now, I’ve been mean­ing to dig into Kamaelia but was largely put off by what I tend to call the “twisted effect”. What this means is that when I go look­ing for libraries and small com­po­nents, I go look­ing for a library — not a “solu­tion”. I also worry about the “once you go in, you must fol­low this par­a­digm” effect. I’m not going to say that these feel­ing con­tinue to be founded, or are com­pletely ratio­nal — after all, I am dig­ging into it, yes? It’s the thought that once you adopt the “one true way of doing things” you’re trapped in that solution/framework “for­ever” — iron­i­cally, I love Django for it’s “con­cep­tual integrity” and full-stack approach. No, I don’t under­stand me either.

Also, as time has pro­gressed, I have found that part of me yearns for a clean and sim­ple to use “frame­work” (note the small “f”) that would help build out a large sys­tem with­out intro­duc­ing a lot of com­plex­ity. Ide­ally, that frame­work would allow me to swap com­po­nents in and out — think of web frame­works like django and tur­bo­gears — in this case, instead of using stock local­ized IPC, I might want to swap in a sim­ple mes­sag­ing pro­to­col (Pyro, XMPP, etc).

It’s also a mat­ter of mar­ket­ing and approach­a­bil­ity — things have improved on both web­sites mind you. Look­ing at Kamaelia’s web­site though, I don’t find it approach­able, as it’s not imme­di­ately clear what the core idea is, or what the dif­fer­ence between Axon (the core) and Kamaelia (the project) is. For exam­ple, if I had one cri­tique, I would say that Axon should become it’s own “project”/library in and of itself, and almost have it’s own web­site. It would be like rip­ping Twisted’s reac­tor out and mak­ing it a com­pletely sep­a­rate library.

Kamaelia, like Twisted, is based on a “sim­ple” core — in this case, it’s the Axon library which has some very sim­ple goals and par­a­digms it seeks to ful­fill. To quote the Axon page:

Axon is a com­po­nent con­cur­rency frame­work. With it you can cre­ate soft­ware “com­po­nents” that can run con­cur­rently with each other. Com­po­nents have “inboxes” and “out­boxes” through with they com­mu­ni­cate with other components.

A com­po­nent may send a mes­sage to one of its out­boxes. If a link­age has been cre­ated from that out­box to another component’s inbox; then that mes­sage will arrive in the inbox of the other com­po­nent. In this way, com­po­nents can send and receive data — allow­ing you to cre­ate sys­tems by link­ing many com­po­nents together.

Each com­po­nent is a micro­process — rather like a thread of exe­cu­tion. A sched­uler takes care of mak­ing sure all micro­processes (and there­fore all com­po­nents) get reg­u­larly exe­cuted. It also looks after putting micro­processes to sleep (when they ask to be) and wak­ing them up (for exam­ple, when some­thing arrives in one of their inboxes).

This by itself is the shin­ing gem of the Kamaelia ecosys­tem — every­thing else is appli­ca­tions or addi­tional util­i­ties built on this sim­ple core. This is where the web­site con­fu­sion comes in — where does “solu­tions built with axon (e.g. kamaelia)” end and Axon begin? The core design (of Axon) is very sim­ple though: build a com­po­nent which com­mu­ni­cates via mes­sage pass­ing.

Mes­sage pass­ing is a rel­a­tively sim­ple con­cept. Com­po­nent A gen­er­ates some work, and then sends it to Com­po­nent (Not A). Mes­sages are han­dled by the receiver and results can be passed (via a mes­sage) to some­one else.

Very, very sim­ple. You can add on lit­tle fac­toids about the fact that mes­sages sent and received are han­dled in asyn­chro­nous fash­ion, mes­sages can be sent locally — or across a wide net­work, etc — but largely those are com­po­nent imple­men­ta­tion details.

Which gets us back to Axon.

Since I’m inter­ested in the core — and not video transcoders — I hit up the Mini­Axon tuto­r­ial here and worked through it — even then, I don’t really think it did Axon com­plete jus­tice. I then jumped into the “How to write new com­po­nents” arti­cle by Michael.

The sec­ond tuto­r­ial, in my hum­ble opin­ion, should be the first arti­cle users are directed to, while it has some pol­ish­ing issues, I found it to really explain what the fruit was going on — and what Axon is.

Read­ing through both of these, you begin to real­ize that Axon is built on the core con­cept of Python gen­er­a­tors and yield­ing con­trol to a sched­uler. For example:

1
2
3
4
5
6
7
8

def sender():
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
   sock.bind((ANY,SENDERPORT))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 255)
   while 1:
      time.sleep(0.5)
      sock.sendto("Hello World", (MCAST_ADDR,MCAST_PORT) );
      yield 1

If you’re famil­iar with python gen­er­a­tors, you know that what this func­tion does is send a mes­sage to a socket, and then hand con­trol back to the main pro­gram. It will do so for­ever until the con­troller exits.

This par­a­digm is key to Axon: com­po­nents send and receive mes­sages via mail­boxes (by far, one of the best descriptions/abstractions I’ve seen for this) — the com­po­nents do the work sent/generated and then put it in the right out­box, and then yield con­trol, cour­tesy of Enhanced Gen­er­a­tors (see PEP 342)

Yes, coroutines/greenlets/tasklets — stop both­er­ing me.

In the tuto­r­ial I linked above, Michael takes the very sim­ple net­work script and ports it to Axon. Here’s my sim­ple exper­i­ment that drops the net­work­ing code and cuts to the mail­box sys­tem. In this case, all I want to do is send and receive the lyrics to the meow mix com­mer­cial, forever:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

import Axon
 
LYRICS="I want chicken I want liver Meow Mix Meow Mix Please Deliver."
 
class Producer(Axon.Component.component):
    def main(self):
        while 1:
            self.send(LYRICS, "outbox")
            yield 1
 
class Sender(Axon.Component.component):
    def __init__(self):
        self.__super.__init__()
 
    def main(self):
        while 1:
            if self.dataReady("inbox"):
                message = self.recv()
                self.send(message, "outbox")
            yield 1
 
class Receiver(Axon.Component.component):
    def __init__(self):
        self.__super.__init__()
 
    def main(self):
        while 1:
            message = self.recv()
            print message
            yield 1
 
def tests(): 
    from Axon.Scheduler import scheduler 
 
    class testComponent(Axon.Component.component): 
        def main(self): 
            producer= Producer()
            sender = Sender()
            receiver = Receiver() 
 
            self.link((producer, "outbox"), (sender, "inbox"))
            self.link((sender, "outbox"), (receiver, "inbox"))
            self.addChildren(producer, sender, receiver)
            yield Axon.Ipc.newComponent(*(self.children)) 
            while 1: 
                self.pause() 
                yield 1
 
    harness = testComponent() 
    harness.activate() 
    scheduler.run.runThreads(slowmo=0.1) 
 
if __name__=="__main__": 
    tests()

Now, this uses knowl­edge from both tuto­ri­als, and the Axon.Component doc­u­men­ta­tion. The com­po­nent doc­u­men­ta­tion can be hard to find, and not so easy to nav­i­gate too.

If we break one of the classes/components down and look at the “magic” pro­vided by the com­po­nent sub­class, it gets clearer — in this case, I’ve “added” back in the meth­ods from the com­po­nent super­class, minus the doc strings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class Sender(Axon.Component.component):
    # First, subclass the Axon Component class, this provides us with the
    # basic inbox/outbox static members that look like this:
    Inboxes = { "inbox"   : "Send the FOO objects to here",
                "control" : "NOT USED",
              }
    Outboxes = { "outbox" : "Emits BAA objects from here",
                 "signal" : "NOT USED",
               }
 
    def __init__(self):
        self.__super.__init__()
 
   def recv(self, boxname="inbox"):
      # returns the first piece of data in the requested inbox.
 
      return self.inboxes[boxname].pop(0)
 
   def send(self, message, boxname="outbox"):
      # appends message to the requested outbox.
 
      self.outboxes[boxname].append(message)
 
   def dataReady(self,boxname="inbox"):
      # Returns true if data is available in the requested inbox.
 
      return self.inboxes[boxname].local_len()
 
    def main(self):
        while 1:
            if self.dataReady("inbox"):
                message = self.recv()
                self.send(message, "outbox")
            yield 1

I think this makes it abun­dantly clear what’s hap­pen­ing with the method calls on this class. Now, there’s the addi­tional magic of the new tests method — in which we defined a new com­po­nent, which was actu­ally a com­po­nent con­tain­ing and link­ing the pipelines (con­nec­tions between mail­boxes) between the other components.

Now, the Axon.Ipc.* docs aren’t the most help­ful — in our case, we called:

1
2
3
4

    self.link((producer, "outbox"), (sender, "inbox"))
    self.link((sender, "outbox"), (receiver, "inbox"))
    self.addChildren(producer, sender, receiver)
    yield Axon.Ipc.newComponent(*(self.children))

Within the main method of a com­po­nent. We need to look at the link method on the superclass:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

   def link(self, source,sink,*optionalargs, **kwoptionalargs):
      """\
      Creates a linkage from one inbox/outbox to another.
 
      -- source  - a tuple (component, boxname) of where the link should start  
                   from
      -- sink    - a tuple (component, boxname) of where the link should go to
 
      Other optional arguments:
 
      - passthrough=0  - (the default) link goes from an outbox to an inbox
      - passthrough=1  - the link goes from an inbox to another inbox
      - passthrough=2  - the link goes from an outbox to another outbox
 
      See Axon.Postoffice.postoffice.link() for more information.
      """
      return self.postoffice.link(source, sink, *optionalargs, \
                                  **kwoptionalargs)

This leads us to the Postof­fice class which actu­ally con­structs and tracks the links between the com­po­nents. Here there be dragons.

So, add­Chil­dren just reg­is­ters all of the passed in com­po­nent instances as chil­dren of the newly con­structed com­po­nents (com­po­nents, all the way down), and then we yield our­self — if you added a ‘print har­ness’ you’d see:

1
2

Component __main__.testComponent_5 [ inboxes : {'control': [], 'inbox': []}
outboxes : {'outbox': <>, 'signal': <>}

This means we’re get­ting a com­po­nent back, and then call­ing .acti­vate on it — .acti­vate is actu­ally a method on the Axon.Microprocess.microprocess class. In our case, it (it being acti­vate) is sim­ply reg­is­ter­ing our test com­po­nent (which con­tains all of the chil­dren) with the default scheduler.

At which point, we call scheduler.run.runThreads jazz hands.

I dived into some of the inter­nals here, I ended up hav­ing to sup­ple­ment the doc­u­men­ta­tion with pour­ing through the code — but I per­son­ally think it helps clear things up to remove some of the magic and show what is actu­ally occur­ring. Most of the time, it seems you sim­ply won’t care — and instead you’d just make and reg­is­ter your happy com­po­nent and be on your way.

To some­what sum­ma­rize what we’re lookin at — a com­po­nent that sub­classes the default Axon.Component uses gen­er­a­tors to yield con­trol back and forth, pass­ing messages/work to and from each other via the very clever mailbox/postoffice metaphor.

Now, the inter­est­ing thing, once you start dig­ging through things is that your com­po­nent isn’t a really a Thread — if you wanted to make sure each com­po­nent was in its own thread, you might instead sub­class Axon.ThreadedComponent.

Sub­class­ing this new class looks mighty close to what we did before but instead Uses threads and queues for the mes­sage pass­ing. Instead of yield, you just run, and the recv/send meth­ods are backed by queues. Ahh, deli­cious non judg­men­tal queues.

In any case, Kamaelia — via Axon, is a very nice abstrac­tion on top of a very sim­ple con­cept — mes­sage pass­ing for con­cur­rency. The fact that you can quickly build up a series of com­po­nents which pass work back and forth via some sort of com­mu­ni­ca­tions sys­tem and not have to worry about the under­ly­ing nuances/organization is quite nice.

One of the things Michael Sparks and I have talked about is adding some level of mul­ti­pro­cess­ing sup­port for Kamaelia — this would actu­ally be insanely easy if I used Axon.ThreadedComponent as the tem­plate, but instead used multiprocessing.Queue and multiprocessing.Process as the back end.

Kamaelia itself, is really a series of exam­ple components/applications which build on a core (Axon), but you do not need Kamaelia to use Axon effec­tively. In fact, just on a whim, I decided to whip up a dirty http load tool in Kamaelia. You can see it here.

It’s really a hack job — all I did was build off the meowmix demo, swapped in the threaded com­po­nent and hacked it around a bit. One thing I’d like to know is how to pass in a dynamic num­ber of clients so that I could cre­ate the out­boxes dynam­i­cally in the pro­ducer — there wasn’t any­thing clear in the docs to allow me to do this. Also, it doesn’t shut down.

I’m going to keep hack­ing around with Axon, it’s pretty neat. Inter­est­ing things I’d like to poke around in:

• Replace the under­ly­ing IPC mech­a­nism with multiprocessing.Pipe/Pyro/posix_ipc
• hack on the mp ver­sion of the com­po­nent backend
• get a multi-system script run­ning and com­mu­ni­cat­ing work­loads across the LAN

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing...