Circuits: event driven components.

by jesse in , , ,


Next up in the GBLOSTR (great big list of stuff to review) is the Circuits library by James Mills (here and here) I'm familiar only with James Mills' posts on python-list, but more recently, I know he's been working on getting some level of multiprocessing into circuits - circuits was already on my research list, but I bumped it up on the queue because we started chatting. Let me state this: these are slightly cleaned up versions of my notes as I am learning these modules - some of them have higher learning curves for me than others.

Circuits is, well - an event based "framework" (again, note the small f) based around the concept of Components (big C!) consuming/reacting and in turn generating events - all asynchronously.

James' goals seem pretty simple - build something with no external dependencies, that's compact (I should say, the core.py is < 500 lines) and that makes it easy to build scalable messaging based (event based) systems. Did he succeed - don't know! Let's dig in.

Note:Given his site is/was down at the time of this writing, I simply cloned the hg repo (here) and used that for all the code and documentation, which is why I am going to sparse on direct links to the documentation. Also note, that pulling from tip introduces dependencies on python 2.6.

Starting with the quickstart - always a good place to jump to, James outlines the very basics - as in, the very, very basics via a simple code example, which I have paraphrased, renamed the objects of and... you get the picture:

from circuits import listener, Event, Component, Manager

class AnEvent(Event):
    "AnEvent Event"

class TheComponent(Component):

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

def main():
    manager = Manager()

    thecomponent = TheComponent()

    manager += thecomponent

    for i in range(10):
        manager.push(AnEvent(), "anevent")

    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break

if __name__ == "__main__":
    main()

And of course, if you run this - you see "hello, I got an event" print 10 times. Simple! But here there be magic, so let's break it down. First, the events.

Here, we subclass and generate a new event:

class AnEvent(Event):
    "AnEvent Event"

If you look in core.py, you'll find that Event is a container - there's some magic here at first glance - but really all this does (see __new__) is construct a new object of the passed in class, and chain any arguments into attributes. The rest is just accessing and comparison - for example:

>>> from circuits import Event          
>>> class x(Event):
... 	"an x event"
... 
>>> y = x(1,2,3,4,arg_one='foo',arg_two='bar')
>>> print y

>>> 

This means, that you can pass in any number of positional arguments and keyword arguments, and they'll be packed into your event. More on this later.

Moving on, the next thing we define is TheComponent:

class TheComponent(Component):

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The parent class of this object is a little bit more magical. James doesn't seem shy of the metaprogramming:

class Component(BaseComponent):

    __metaclass__ = HandlersType

Nuts. Right above it though is the definition of BaseComponent which is slightly simpler but reveals something interesting - each component can be passed in a channel, and the default is None (more on channels in a bit). Additionally, we see BaseComponent is a subclass of a Manager. Components are managers, and managers are components.

Finkle is einhorn, einhorn is Finkle!

The metaclass does magical object construction stuff and method management/etc. Dragons.

So, the superclass has two addition methods - register() and unregister() which control the component's registration with the manager (more later). Otherwise, it doesn't have a lot else, except for the on_anevent() method we've added.

And made fancy with a decorator (mandatory pieces of flair)!

This code bit:

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

Essentially means "this method reacts to this event" - so, let's look at the listener decorator a bit. The decorator is in core.py, and the docstring explains a lot more about the various arguments the decorator can take, but not very clearly. I guess you had to be there.

The one thing you notice though, is that the decorator adds some method attributes to your method - things like f.type, f.target, etc. And then something called f.br, the use of which isn't clear to me right now. You can however have a single handler listen for multiple events:

class TheComponent(Component):

    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"

note: James explained the .br attribute:

handler.br is used in the BaseComponent's send(...), and iter(...). It is used to determine which "branch" to follow. The args and kwargs of an Event are intelligently applied to an event handler depending on the signature of the event handler. (It took some time to get right - but basically it means you can't go wrong!)

In short - we now have a registered listener for an event named "anevent" - when an event is pushed (more on this later). More recently James altered it so that if you define a component, any method which doesn't start with a _ in the name will explicitly become an event handler of "listener" type and listens on a channel which is based off the method name.

We can refactor the original code like this:

class TheComponent(Component):

    def anevent(self):
        print "hello, I got an event"

And it works just as well - which is a nice improvement. Here's the note from James on this:

Basically, the new HandlersType (metclass) means that every method defined in a sub-classes Component that have not been previously defined as event handlers with the @listener decorator or do not start with a _, are automagically turned into event handlers listening on a channel that is the name of the method.

The use of the @listener decorator is still required for:

  • Defining filters
  • Defining event handlers listening on multiple channels.
  • Defining event handlers listening on foreign targets.

Next up is the main function - here we instantiate a new Manager object, which if you look at the code of this, you'll notice right off the bat that there's a startling number of __ method overloading. Which explains this line:

    manager += thecomponent

This is handled by the manager's __iadd__ method:

    def __iadd__(self, y):
        y.register(self.manager)
        if hasattr(y, "registered"):
            y.registered()
        return self

This means that when you append it onto the manager (something more explicit would be nice - brevity and terseness be damned) it calls the register function on the component. Reading through that method is actually quite telling as it introspects the current object (a component) and extracts the callable methods and then pulls out the handlers/channels for a given event and calls .add on the manager. A given component may have multiple listeners for a given event inside of a component:

class TheComponent(Component):

    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"

    def anevent(self):
        print "hello, I got an event"

These handlers are all registered with the manager. Now, something I want to clear up is that when you declare:

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The "anevent" defines the channel that listener listens on. It's easy to say "listens for events" - but really what this is is a subscription to a channel. This means you can define a listener with, well - no event:

    @listener()
    def all(self):
        print "i listen for everything"

It's like having all the premium channels.

Back to the manager though (the above was a light bulb going off in my head) - next we see that we "push" events into the manager. This is actually pretty simple:

    def push(self, event, channel, target=None):
        """E.push(event, channel, target=None) -> None

        Push the given event onto the given channel.
        This will queue the event up to be processed later
        by flushEvents. If target is given, the event will
        be queued for processing by the component given by
        target.
        """

        if self.manager == self:
            self._queue.append((event, channel, target))
        else:
            self.manager.push(event, channel, target)

This means we can push events into the manager, and target them at a specific component. The definition of an event in the case of a listener is actually the creation of a channel.

The push method exposes something else - it seems possible to create a manager which is actually a pointer to another manager (a proxy) alas, I don't see how to do that. Hooray!

So, we can create new events and push them into the manager which pipes them off to the channels who have assigned listeners. Easy! We flush the manager's queue so all events are pushed to the components, and we're done.

So what can we use these basics for? Easy - building components which stack on top of each other which contain other components which subscribe to a given event. In the examples directory, James has done an excellent job showcasing a lot of problems which he solves using the core circuits library. In fact, the examples explain a lot more about how things work than the core code itself.

One of the questions I had in working though all of this, is what happens if you define a general event - can the handler gain access to arguments within the event? Does it need to? Is it simply sufficient for a listener on a given channel to know the name/type of an event and react to it?

The answer is, well, yesandno. Let's say we do this:

from circuits import listener, Event, Component, Manager

class AnEvent(Event):
    "AnEvent Event"

class TheComponent(Component):

    def anevent(self, *args, **kwargs):
        print args, kwargs

def main():
    manager = Manager()

    thecomponent = TheComponent()

    manager += thecomponent

    for i in range(10):
        manager.push(AnEvent(1,2,3, no='yes', yes=False), "anevent")

    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break

if __name__ == "__main__":
    main()

You would see:

(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}

Awesome. The arguments the event is created with are passed directly into the listener for that, so as long as you have the right signature on the method, you should be golden.

Which leaves us with a basic question - if messages have to be pushed into the manager on a given channel, how do we build something which is an event generator? In other words - how do we make something which "listens" and then generates the matching events.

To know that - we look in the lib/ directory (which the examples make prodigious use of) and we'll focus on io.py which listens on stdin:

from circuits import listener, Event, Component, Manager
from circuits.lib.io import Stdin

class DogBot(Stdin):

    def read(self, *args, **kwargs):
        print args, kwargs

def main():
    manager = Manager()
    dog = DogBot()
    manager += dog

    while True:
        try:
            manager.flush()
            dog.poll()
        except KeyboardInterrupt:
            break

if __name__ == "__main__":
    main()

All this does is create a new component which listens for the read event that Stdin generates and then prints off what we see coming in off the command line:

thumper:circuits jesse$ python2.6 dog.py 
a line is one argument
('a line is one argument\n',) {}

That means the read event needs to parse the passed in line and then (re)issue and event for some other listener to manage. Let's define some addition events:

class Sit(Event):
    """ Sit event, no args """

class Say(Event):
    """ Speak event
     args: what to say
    """

class DogBot(Stdin):

    def read(self, *args, **kwargs):
        command = args[0].strip('\n').split()
        if not command:
            return
        if command[0] == 'sit':
            self.push(Sit(), 'sit', self.channel)
        elif command[0] == 'say':
            self.push(Say(' '.join(command[1:])), 'say', self.channel)

    def sit(self):
        print 'i am now sitting'

    def say(self, words):
        print '%s' % words

Rudimentary - but you get the idea. For networking, you'd need to bind the socket and then read off/generate the events - this is largely covered by the sockets.py module in lib, as well as irc, webserver and smtp listeners.

All in all - it's pretty simple to construct components, it could be made a bit easier with less metaprogramming and more, well, methods but a great deal more documentation could help too. I'm not too fond of too much metaprogramming - I tend to think it makes code rather unapproachable in general.

The (event/channel)/subscription model used here is nice as well - you can easily create your own little asynchronous network daemon or something as simple as what I did above very quickly (once you know what's going on). It is obviously still evolving - James is putting a lot of work into it. If you're looking for a compact little library, this would be good to check out.