jessenoller.com

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on python-list, but more recently, I know he’s been work­ing on get­ting some level of mul­ti­pro­cess­ing into cir­cuits — cir­cuits was already on my research list, but I bumped it up on the queue because we started chatting.

Let me state this: these are slightly cleaned up ver­sions of my notes as I am learn­ing these mod­ules — some of them have higher learn­ing curves for me than others.

Cir­cuits is, well — an event based “frame­work” (again, note the small f) based around the con­cept of Com­po­nents (big C!) consuming/reacting and in turn gen­er­at­ing events — all asynchronously.

James’ goals seem pretty sim­ple — build some­thing with no exter­nal depen­den­cies, that’s com­pact (I should say, the core.py is < 500 lines) and that makes it easy to build scal­able mes­sag­ing based (event based) sys­tems. Did he suc­ceed — don’t know! Let’s dig in.

Note:Given his site is/was down at the time of this writ­ing, I sim­ply cloned the hg repo (here) and used that for all the code and doc­u­men­ta­tion, which is why I am going to sparse on direct links to the doc­u­men­ta­tion. Also note, that pulling from tip intro­duces depen­den­cies on python 2.6.

Start­ing with the quick­start — always a good place to jump to, James out­lines the very basics — as in, the very, very basics via a sim­ple code exam­ple, which I have para­phrased, renamed the objects of and… you get the picture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

from circuits import listener, Event, Component, Manager
 
class AnEvent(Event):
    "AnEvent Event"
 
class TheComponent(Component):
 
    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"
 
def main():
    manager = Manager()
 
    thecomponent = TheComponent()
 
    manager += thecomponent
 
    for i in range(10):
        manager.push(AnEvent(), "anevent")
 
    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

And of course, if you run this — you see “hello, I got an event” print 10 times. Sim­ple! But here there be magic, so let’s break it down. First, the events.

Here, we sub­class and gen­er­ate a new event:

1
2

class AnEvent(Event):
    "AnEvent Event"

If you look in core.py, you’ll find that Event is a con­tainer — there’s some magic here at first glance — but really all this does (see __new__) is con­struct a new object of the passed in class, and chain any argu­ments into attrib­utes. The rest is just access­ing and com­par­i­son — for example:

1
2
3
4
5
6
7
8

>>> from circuits import Event          
>>> class x(Event):
... 	"an x event"
... 
>>> y = x(1,2,3,4,arg_one='foo',arg_two='bar')
>>> print y
<x/ (1, 2, 3, 4, arg_two=bar, arg_one=foo)>
>>>

This means, that you can pass in any num­ber of posi­tional argu­ments and key­word argu­ments, and they’ll be packed into your event. More on this later.

Mov­ing on, the next thing we define is TheComponent:

1
2
3
4
5

class TheComponent(Component):
 
    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The par­ent class of this object is a lit­tle bit more mag­i­cal. James doesn’t seem shy of the metaprogramming:

1
2
3

class Component(BaseComponent):
 
    __metaclass__ = HandlersType

Nuts. Right above it though is the def­i­n­i­tion of BaseC­om­po­nent which is slightly sim­pler but reveals some­thing inter­est­ing — each com­po­nent can be passed in a chan­nel, and the default is None (more on chan­nels in a bit). Addi­tion­ally, we see BaseC­om­po­nent is a sub­class of a Man­ager. Com­po­nents are man­agers, and man­agers are components.

Fin­kle is ein­horn, ein­horn is Finkle!

The meta­class does mag­i­cal object con­struc­tion stuff and method management/etc. Dragons.

So, the super­class has two addi­tion meth­ods — reg­is­ter() and unreg­is­ter() which con­trol the component’s reg­is­tra­tion with the man­ager (more later). Oth­er­wise, it doesn’t have a lot else, except for the on_anevent() method we’ve added.

And made fancy with a dec­o­ra­tor (manda­tory pieces of flair)!

This code bit:

1
2
3

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

Essen­tially means “this method reacts to this event” — so, let’s look at the lis­tener dec­o­ra­tor a bit. The dec­o­ra­tor is in core.py, and the doc­string explains a lot more about the var­i­ous argu­ments the dec­o­ra­tor can take, but not very clearly. I guess you had to be there.

The one thing you notice though, is that the dec­o­ra­tor adds some method attrib­utes to your method — things like f.type, f.target, etc. And then some­thing called f.br, the use of which isn’t clear to me right now. You can how­ever have a sin­gle han­dler lis­ten for mul­ti­ple events:

1
2
3
4
5

class TheComponent(Component):
 
    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"

note: James explained the .br attribute:

handler.br is used in the BaseComponent’s send(…), and iter(…). It is used to deter­mine which “branch” to fol­low. The args and kwargs of an Event are intel­li­gently applied to an event han­dler depend­ing on the sig­na­ture of the event han­dler. (It took some time to get right — but basi­cally it means you can’t go wrong!)

In short — we now have a reg­is­tered lis­tener for an event named “anevent” — when an event is pushed (more on this later). More recently James altered it so that if you define a com­po­nent, any method which doesn’t start with a _ in the name will explic­itly become an event han­dler of “lis­tener” type and lis­tens on a chan­nel which is based off the method name.

We can refac­tor the orig­i­nal code like this:

1
2
3
4

class TheComponent(Component):
 
    def anevent(self):
        print "hello, I got an event"

And it works just as well — which is a nice improve­ment. Here’s the note from James on this:

Basi­cally, the new Han­dler­sType (met­class) means that every method defined in a sub-classes Com­po­nent that have not been pre­vi­ously defined as event han­dlers with the @listener dec­o­ra­tor or do not start with a _, are automag­i­cally turned into event han­dlers lis­ten­ing on a chan­nel that is the name of the method.

The use of the @listener dec­o­ra­tor is still required for:

• Defin­ing filters
• Defin­ing event han­dlers lis­ten­ing on mul­ti­ple channels.
• Defin­ing event han­dlers lis­ten­ing on for­eign targets.

Next up is the main func­tion — here we instan­ti­ate a new Man­ager object, which if you look at the code of this, you’ll notice right off the bat that there’s a star­tling num­ber of __ method over­load­ing. Which explains this line:

1

    manager += thecomponent

This is han­dled by the manager’s __iadd__ method:

1
2
3
4
5

    def __iadd__(self, y):
        y.register(self.manager)
        if hasattr(y, "registered"):
            y.registered()
        return self

This means that when you append it onto the man­ager (some­thing more explicit would be nice — brevity and terse­ness be damned) it calls the reg­is­ter func­tion on the com­po­nent. Read­ing through that method is actu­ally quite telling as it intro­spects the cur­rent object (a com­po­nent) and extracts the callable meth­ods and then pulls out the handlers/channels for a given event and calls .add on the man­ager. A given com­po­nent may have mul­ti­ple lis­ten­ers for a given event inside of a component:

1
2
3
4
5
6
7
8

class TheComponent(Component):
 
    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"
 
    def anevent(self):
        print "hello, I got an event"

These han­dlers are all reg­is­tered with the man­ager. Now, some­thing I want to clear up is that when you declare:

1
2
3

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The “anevent” defines the chan­nel that lis­tener lis­tens on. It’s easy to say “lis­tens for events” — but really what this is is a sub­scrip­tion to a chan­nel. This means you can define a lis­tener with, well — no event:

1
2
3

    @listener()
    def all(self):
        print "i listen for everything"

It’s like hav­ing all the pre­mium channels.

Back to the man­ager though (the above was a light bulb going off in my head) — next we see that we “push” events into the man­ager. This is actu­ally pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

    def push(self, event, channel, target=None):
        """E.push(event, channel, target=None) -> None
 
        Push the given event onto the given channel.
        This will queue the event up to be processed later
        by flushEvents. If target is given, the event will
        be queued for processing by the component given by
        target.
        """
 
        if self.manager == self:
            self._queue.append((event, channel, target))
        else:
            self.manager.push(event, channel, target)

This means we can push events into the man­ager, and tar­get them at a spe­cific com­po­nent. The def­i­n­i­tion of an event in the case of a lis­tener is actu­ally the cre­ation of a chan­nel.

The push method exposes some­thing else — it seems pos­si­ble to cre­ate a man­ager which is actu­ally a pointer to another man­ager (a proxy) alas, I don’t see how to do that. Hooray!

So, we can cre­ate new events and push them into the man­ager which pipes them off to the chan­nels who have assigned lis­ten­ers. Easy! We flush the manager’s queue so all events are pushed to the com­po­nents, and we’re done.

So what can we use these basics for? Easy — build­ing com­po­nents which stack on top of each other which con­tain other com­po­nents which sub­scribe to a given event. In the exam­ples direc­tory, James has done an excel­lent job show­cas­ing a lot of prob­lems which he solves using the core cir­cuits library. In fact, the exam­ples explain a lot more about how things work than the core code itself.

One of the ques­tions I had in work­ing though all of this, is what hap­pens if you define a gen­eral event — can the han­dler gain access to argu­ments within the event? Does it need to? Is it sim­ply suf­fi­cient for a lis­tener on a given chan­nel to know the name/type of an event and react to it?

The answer is, well, yesandno. Let’s say we do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

from circuits import listener, Event, Component, Manager
 
class AnEvent(Event):
    "AnEvent Event"
 
class TheComponent(Component):
 
    def anevent(self, *args, **kwargs):
        print args, kwargs
 
def main():
    manager = Manager()
 
    thecomponent = TheComponent()
 
    manager += thecomponent
 
    for i in range(10):
        manager.push(AnEvent(1,2,3, no='yes', yes=False), "anevent")
 
    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

You would see:

1
2
3
4
5
6
7
8
9
10

(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}

Awe­some. The argu­ments the event is cre­ated with are passed directly into the lis­tener for that, so as long as you have the right sig­na­ture on the method, you should be golden.

Which leaves us with a basic ques­tion — if mes­sages have to be pushed into the man­ager on a given chan­nel, how do we build some­thing which is an event gen­er­a­tor? In other words — how do we make some­thing which “lis­tens” and then gen­er­ates the match­ing events.

To know that — we look in the lib/ direc­tory (which the exam­ples make prodi­gious use of) and we’ll focus on io.py which lis­tens on stdin:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

from circuits import listener, Event, Component, Manager
from circuits.lib.io import Stdin
 
class DogBot(Stdin):
 
    def read(self, *args, **kwargs):
        print args, kwargs
 
def main():
    manager = Manager()
    dog = DogBot()
    manager += dog
 
    while True:
        try:
            manager.flush()
            dog.poll()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

All this does is cre­ate a new com­po­nent which lis­tens for the read event that Stdin gen­er­ates and then prints off what we see com­ing in off the com­mand line:

1
2
3

thumper:circuits jesse$ python2.6 dog.py 
a line is one argument
('a line is one argument\n',) {}

That means the read event needs to parse the passed in line and then (re)issue and event for some other lis­tener to man­age. Let’s define some addi­tion events:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

class Sit(Event):
    """ Sit event, no args """
 
class Say(Event):
    """ Speak event
     args: what to say
    """
 
class DogBot(Stdin):
 
    def read(self, *args, **kwargs):
        command = args[0].strip('\n').split()
        if not command:
            return
        if command[0] == 'sit':
            self.push(Sit(), 'sit', self.channel)
        elif command[0] == 'say':
            self.push(Say(' '.join(command[1:])), 'say', self.channel)
 
    def sit(self):
        print 'i am now sitting'
 
    def say(self, words):
        print '<dog>%s</dog>' % words

Rudi­men­tary — but you get the idea. For net­work­ing, you’d need to bind the socket and then read off/generate the events — this is largely cov­ered by the sockets.py mod­ule in lib, as well as irc, web­server and smtp listeners.

All in all — it’s pretty sim­ple to con­struct com­po­nents, it could be made a bit eas­ier with less metapro­gram­ming and more, well, meth­ods but a great deal more doc­u­men­ta­tion could help too. I’m not too fond of too much metapro­gram­ming — I tend to think it makes code rather unap­proach­able in general.

The (event/channel)/subscription model used here is nice as well — you can eas­ily cre­ate your own lit­tle asyn­chro­nous net­work dae­mon or some­thing as sim­ple as what I did above very quickly (once you know what’s going on). It is obvi­ously still evolv­ing — James is putting a lot of work into it. If you’re look­ing for a com­pact lit­tle library, this would be good to check out.

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on...