There’s been some recent discussion about DjangoCon(.eu | .us) and whether or not speakers should have to pay for admission as well — see Chris Wanstrath’s (of Github) tweets (here and here) and this Convore thread for examples. Obviously, as PyCon is the “big dog” so to speak for Python conferences, everyone looks to “us” for a model to work from, or how we manage things. I’ve seen a lot of poop slung towards the DjangoCon organizers, mainly due to a lack of knowing “why” certain policies (such as “Everyone Pays”) exist for DjangoCon, PyCon, and other conferences.

As co-chair and program committee chair last year, and program committee chair the year before, and now chair for the next two years — I figured it might be good to take a moment to explain the rationale behind PyCon’s approach — as well as some statistics about the budget. I’m not going to state that this policy is perfect; nor that it won’t be changed; I also will not release the budget publicly — I don’t think giving everyone a spreadsheet without the context of the hundreds of man hours of work that go into it is useful, at all.

Important Note: PyCon is organized and managed by the Python Software Foundation — this means that, as part of being a 501c3 charity, some of the financials from past PyCon is available as part of publicly accessible financial documents of the foundation. You can find those on the PSF’s site.

The same reasoning may not apply to a conference that is organized by a commercial entity or is done for profit. OSCON is a commercial conference, so having speakers get in free is generally expected. DjangoCon.us is in the middle — it is organized for profit by a commercial entity, but it also contributes heavily back to the Django Software Foundation. DjangoCon.eu is managed differently as well.

Policy and Financial Aid

First, what is the “policy” — well, first, it’s not a policy per-se; it’s part policy, and part tradition. The policy is “Everyone pays for a ticket” — attendees, tutorial presenters, organizers (meaning: all volunteers, including the chair) and sponsors — everyone pays their way. Yes, this means Van and myself both paid for our own tickets to attend. Everyone is treated as equal — sponsors get free admissions as part of conference sponsorship packages, and keynote speakers may be provided with either compensation or admission depending on the negotiated deal (some keynote speakers cost money, for example).

Now; an interesting aspect of this is that PyCon, as a conference, offers a very generous financial aid program — this means that some attendees, speakers, tutorial presenters, etc have some, or in rare cases, all of their expenses such as flight, hotel and admission provided to them from the PyCon budget. PyCon goes out of it’s way to encourage people to apply for financial aid — even if we can’t cover all of your expenses, we will give you free admission based on need. The FA application process is simple, and straightforward. It’s also very liberal — the only caveat is that speakers at the conference “get bumped to the top” of the applications so that we don’t lose a good talk because of financial need. We also don’t ban anyone from applying (for example, I needed assistance in 2010 even as the PC chair).

We use the FA budget to not only help the “normal” attendees of the conference, we also keep an eye towards diversity — for example in 2010 we had a specific grant program (funded by Google) for women to attend PyCon — which they did in amazing numbers! We also try to help more people with less money than less people with more money — we want to spread assistance out as much as possible. This is why FA requires room-sharing at the conference hotel, this is why we may only cover part of a given applicant’s costs — we want to help more people.

As you can guess — since financial aid is a part of the budget, we allocate a set amount of money to this — but, without a solid budget (meaning, admission ticket sales, sponsorships, etc) we can’t have it — it’s actually a dice roll. We can guess at how much FA we can carve out of the budget at the start of the process, and we will award that amount — but that means if we have low attendance, don’t book enough rooms, or our catering costs spike, or lack sponsors, the conference can lose not insignificant amounts of money helping people. The rub being, you don’t know for sure if this will happen until the conference actually happens.

So, how does “everybody pays” play into this? In two ways. First, it helps hedge our risk, institutionally speaking. PyCon had a good/bad year in 2009 — good for the attendees, but horrendously bad financially. We made various commitments in 2008 just as the markets were peaking, and we lost a lot of money when everything went south. Since then, it has been our budget policy to make sure that our revenues will cover our hard costs even if there is a 15% dip in attendance and a 35% dip in sponsorship.

Second, you may have guessed that the “everybody pays” policy allows us to pad the financial aid budget. This has the direct benefit of bringing more people to the conference, increasing the diversity of people attending, getting a larger proportion of the community together, etc.

I’m happy to say, that on most years, we very happily break our financial aid budget — meaning, if we have a positive outlook, we will gladly overspend on financial aid and take less “profit ” for the conference. The point of the conference is the community, it’s not about the conference! We help the community as much as we can by helping to cover the costs of people who would not otherwise be able to attend.

History and Tradition

Once you start organizing a conference you start to rapidly realize how costly things can get — and if you don’t have sponsors with giant bags of money, or the conference is not being run for profit (such as OSCON), you can quickly get into trouble if you don’t have a policy in place that will ensure financial viability. PyCon came from very small, humble beginnings but has grown year over year consistently — maintaining the “everyone helps, and everyone pays” tradition originally set forth.

It is my understanding that the policy/tradition has as much to do with financials as it does with “the feel”. No one is treated specially, we — meaning the community — put together the conference, we are taking the financial risk that comes with it. We are in it together, and that means “everyone helps, and everyone pays” — it is the most equitable solution that helps us maintain fairness and the flavor/feel of the conference. When you realize that even Guido has to buy a ticket (even if it was paid for by Google) or that Van Lindberg had to buy one — it suddenly makes you realize that all of us truly believe in the spirit of “we are all in this”.

Speakers; as a an additional guideline, while always paying for a ticket; always pay at the discounted early bird rate no matter when they register. So yes, we do discount them, but no more than anyone else who registers within the early bird window.

As an aside, the “everybody helps” is also part of what makes PyCon special. When people come to other conferences, they frequently come expecting to be coddled and catered to. When people come to PyCon, we hope they come expecting to contribute. Having everybody contributing is at the core of the best parts of PyCon — from stuffing bags, to lightning talks, to the wonderful Testing in Python BOF.

Hard Costs, Stats and Risk

So what makes up a PyCon budget? Well — Van covered some of it in his wonderful “Behind the Scenes” post for PyCon 2011 — but in more detailed terms of percentages of the total costs for the budget:

• Catering (Food, beverages): 54%
• Networking: 5%
• Audio/Visual and Video Recording: 13%
• Payments to Tutorial Instructors: 6%
• Misc/PR/Sponsors and Administration: 16%
• Swag: 2%
• Financial Aid: 4%

Catering, plus Financial Aid and video recording is 70% of the budget alone. The total value “spent” for financial aid alone well exceeds 50,000$ We spend a massive amount of money for audio/visual and the video recording making all of the talks available on the web (see the python miro community). Tens of thousands of dollars spent on networking.

We could forgo feeding people — but with a conference that runs from 8am to 6pm for 5 days, that would suck — and people needing to leave to get food would be disruptive and ruin the flow. We could forgo recording the videos. We could do a lot of things to cut back and lower the budget numbers so we could give more away (such as comp’ing speaker tickets), but that doesn’t make the risk and costs from the hotel go away.

• Average Revenue per attendee: ~$300
• Average Cost per attendee: ~$465

In total — these costs (for 2011) came to well over 600,000$ — that’s right. Over half a million dollars in hard costs for a conference of 1,380 Pythonistas. Much of the payments, guarantees, deposits, etc have to be put down up front of the actual conference with no guarantee that we will make our sponsorship or attendee numbers. That means, in the case of PyCon — the Python Software Foundation fronts potentially hundreds of thousands of dollars in payments and risk. We also expect these to climb with the move of the conference to Santa Clara.

Conferences are unknowns — you put up a lot of time, energy and money — lots of money — and you hope you hit certain numbers. For example, you make a contract with the hotel that says “we will have X number of people book X number of rooms nights” — if you don’t meet those numbers? The hotel penalizes you a stunning amount of money. You make a contact for catering — saying that you will serve X number of meals — and if you don’t hit that? You get penalized. You put a lot of faith in making numbers on:

• Sponsors — without corporate sponsors, we simply could not exist (at this scale), period.
• Attendee Numbers — You expect X number of attendees, broken up into “corporate, individual and Y” rates — corporate rate tickets are where you make the bulk of registration rates.
• Hotel Rooms — if you don’t hit your contracted numbers, you will take a bath

Do you know what happens if, for some reason — say an economic downturn — you don’t make those numbers? Your conference loses over $230,000, possibly more. That 230k number? Right around the amount PyCon 2009 lost, representing a massive financial loss for the Python Software Foundation. Yes, you try to mitigate the risks by careful planning — if you’re lucky, you get some insurance to cover some of the potential losses. PyCon is actually really lucky in all of this — we’ve grown enough that we get to leverage economies of scale with the contracts we deal with. Small conferences? Not so much — they get harder in the face of failure, and they don’t have the bargaining power we have..

It’s a delicate balancing act, requiring careful planning and a lot of work to drive attendance and sponsorship — negotiations with Hotels, Catering, Unions, are all things that have to go right in order to make a conference that even stands a chance of not losing you massive amounts of money. In the case of PyCon — a massive loss of money (such as 2009) could cause the foundation to go bankrupt and dissolve. As it is, the foundation, already spartan in spending cut back even further to protect itself which represents a net negative for the Python Community.

Where does the profit go?

Alright — I’ve outlined the philosophy, the scale and the risk. Now, you’re probably asking where, if we have it, does any profit from the conference go? Easy — straight back into the community.

Whatever profit — really, money left over, goes to cover the initial costs/risks for PyCon the next year, and the years after — it also goes to the Python Software Foundation that in turn, reinvests that money into the community (see the foundation blog).

Having a healthy positive margin also means we can increase things such as the financial aid budget for the next year, allowing even more Python developers to attend.

Why not run a budget that aims for 0$ +/-?!

Because, that is simply insane — you can aim for minimal profit by reinvesting into things such as financial aid, or other perks for the attendees but only after you have guaranteed you’re going to make your numbers — see the issue? At best, you budget is a highly refined series of educated guesses. At worst? Someone goes bankrupt — at smaller conferences such as DjangoCon.eu, where some costs and contracts are secured by an individual and their credit card?

That person goes broke, because the entities you signed contracts with will get their money.

So, you never, ever aim for a loss, but you also don’t aim for extracting “maximum value” from conference attendees. You aim, you plan, and you hope to provide the best value and experience to attendees and sponsors — you do this through a lot of hard work and sweat. You make sure attendees walk away raving about the conference, you make sure you generate the maximum return on investment for the community.

In Closing

Historically, PyCon has never been aimed at “extracting money” from the community — it could be run a lot more profitably. We could increase ticket rates, reduce FA and walk around in Nascar-esque suits with sponsor logos on them — there’s lots of things that could be done to make PyCon a massive profit center for the PSF. We don’t though — why? Because, that defeats the purpose of a community conference, put together by and for the community.

We take what money we end up with and we sponsor the sprints at PyCon itself, we hold the language and virtual machine summits. The PSF reinvests it into the Python community as a whole (see the sprints project, for example, or the 10,000$ check we gave to PyPy). The money is given out in the forms of developer grants, holding sprints, providing sponsorship to smaller Python conferences all over the world, outreach and education initiatives, etc. No one gets a shiny new car or anything — money that comes from the community conference goes back into the community. You can see how much goes where if you peruse the PSF resolutions page.

PyCon is beginning to also outstrip the ability for a small group of volunteers to put together — more and more we’ve had to bring on commercial entities (whom we have to pay) to assist in organizing or doing specific things (such as registration, or website development). I fully expect PyCon 2012/2013 to quickly hit the attendance cap meaning we’re going to have more scaling issues and higher costs — this is a fundamentally good thing — PyCon, it is felt, can sustain it’s current feel and quality up to a maximum of 1500–2000 attendees.

We do not have the intention of raising that cap.

Instead, we — as the foundation — intend on encouraging smaller more regional conferences such as PyOhio to grow — PyCon will continue to be the “big dog” so to speak — but it being the big dog will allow us to further cultivate and pollenate the entire Python community and ecosystem.

Given all of that — as the PyCon chair for the next two years — and being relatively comfortable that we will again, put on a highly successful conference — I am exploring the idea of steeply discounting the cost of tickets for speakers. I feel that not only is the conference viable once again, but that the speakers do help make the conference what it is — last year we had over 200 talk and tutorial submissions. All of those speakers submitted talks knowing that they would have to pay for a ticket to attend — but given the positive outlook, I’m comfortable in thinking that we will be able to provide accepted speakers with a discount to thank them for their talk.

It’s all about community, and reinvesting in it in the right places, at the right times, and managing your risk. The risks putting on a conference — even a small one (especially a small one) are massive, but the returns — especially when you’re standing on a stage in front of 1380 fellow Python hackers handing a big cardboard check to a fantastic project?

Well, the returns are well worth it.

The best pycon, ever

Yeah, I came out and said it. For me, it was my 4th(?) PyCon, and not to disparage the past, but this one was the best planned, best executed and most fun for me. Sure, it had a few negatives (Chicago being one of them), but by far the pluses outweighed any minuses.

Here’s a summary of things I can remember.

The summits

I got in mid-day wednesday and broke into the VM summit. I was immediately slapped with the unladen-swallow announcement, which made me do a little dance and immediately start pulling down the code. For the most part, since I was past the introduction piece, I sat in back and had interesting discussions with Thomas Wouters, Brett Cannon and others.

Later that night, I sat down and spent some time with unladen swallow — mainly getting LLVM all happy and pulling down the trunk (when I should have been using the released tag). I also spent a fair amount of time massaging my slides for both of my talks. Nothing quite like last minute changes.

The next day was the language summit. Here a bunch of people much smarter than me sat down and discussed the future of the language. I can’t even begin to describe everything that happened — some basic highlights are:

• Some discussions around making python 2.7 be the last release of the 2.x line. I think the general consensus was to continue to use 2.x as a ramp into 3.0, but in general if migration to (and from) 3.0 was faster/easier, then the adoption barrier for 3.x would be greatly lessened. Most of us finally agreed that the 2.x line should end with 2.7.
• On 3.0 — porting issues and pains were discussed, things like 2to3 being slow, lack of incentive (3k being “a better language” notwithstanding) being one of the big issues. Someone (Guido I think) floated the idea that big features only go into the 3.x branch, ergo providing greater upgrade incentive. I don’t think a lot of people disagreed with this and in fact, I took it to heart during the sprints when looking at feature requests for multiprocessing.
• Which brings us to 3to2 — several developers expressed that if we had a tool which could translate python3 code to python2, then they could start writing their frameworks/tools in pure python3 and then back port it. I don’t recall anyone disliking this idea — but no one wanted to get the ball rolling. So I raised my hand. In essence, I volunteered to get the ball rolling, which I have done. Benjamin Peterson will be heading it up primarily — but anyone is free to contribute (and some already have). It’s got potential as a google summer of code project.
• After that we discussed what CPython can do to make integration/testing easier for alternative implementations. We agreed that being able to mark tests as “cpython only” and breaking things in the repository up in such a way to make integration of the standard library into alternative distributions easier. Brett has more on this, but the essential idea is to compartmentalize the standard library out into a separate project within source control, and have other implementation pull/ingrate that. Anything that’s Jython/CPython/etc specific would be marked/stored appropriately. For example, multiprocessing as it exists today is a CPython artifact, and not needed for say, Jython.
• And then there was the packaging discussion. Rather then rehash everything that was discussed, I’ll simply mention that I proposed a much simpler approach. Specifically, I noted that we should simplify the core of distutils, pulling in setuptools feature where appropriate, offer plugin/hooks where appropriate, standardize on a simple, extensible metadata format and remove any of the “non core” bells and whistles from distutils. In essence my argument was that things like RPM building, easy_install, virtualenv, and so on simply do not belong in core python, rather they belong outside of the core, where consumers/developers can tailor solutions to suit their needs. Ultimately disutils can help them, and make the APIs/Plugins/Metadata standard but it should not be a fully fledged “package manager” or build utility.

Ultimately, the summits were a ton of fun, and being lucky enough to be invited was really great for me. At the end I was worried about people being tired of my chirping up, but it was resoundingly a success.

The conference

This is where things begin to blur — hopping in between hotels, lightning talks, hallway discussions — it was insane. Add to that my fear/stress about both of my talks, and I simply can’t fit everything that happened in my brain.

I think I started the day in “How to Give a Python Talk” by AMK — which was great, if disheartening simply due to the fact that I had made some of the mistakes he pointed out in my talks. Ergo more slide hacking. After that it was hallway discussions and hitting up “How Python is developed” by Brett — I was going to heckle him, but I played nice.

Then there was the Python VMs panel, which I swapped out of and into the “Building an Automated QA Infrastructure using Open-Source Python Tools” — which can be summed up as “Yay buildbot” (I’m hard to impress, I’ve built a few of these by now). I then hit up the Twisted/AMPQ talk but ducked out early to go prep for my talk.

Then, it was my multiprocessing talk time. You can see the full video here, I think it went well, and feedback has been positive. Given I’ve done variations on this talk elsewhere, I was a little more confident in myself. I think I could have slowed down in some parts, and stopped fidgeting as well as a few other nits. I had some great conversations with users of the package as well as people new to it which pretty much absorbed all my time until dinner. I think I hit up the lightning talks — but I can’t remember.

The next day, was the now infamous LINDBERG’D lightning talk (video here) — for those who don’t know — Yes, Van knew — he even wrote the legal disclaimer I have in my slides. Basically, Brett and I had stayed up until what — 1am one night laughing about this idea of sending Van (who isn’t very threatening) after commenters on the internet. The day after we stayed up laughing at this (much to the dismay of our neighbors) I hit Van up for some text and let him in on it. No, it wasn’t serious (people asked). Yes, I was also voted in with a pile of other people into the PSF. Good times.

Afterward, the guidonote and then know I saw the state of django talk, as well as Jack’s “Class Decorators: Radically Simple” — which was fantastic (although I saw a variation at the boston python meetup the week before). I dropped in on the ORM panel, and the GAE talk (which was disappointing). Then I hit Bob’s “Drop ACID and think about data” which was awesome. I again ducked out to double check my stuff for my distributed systems talk. Also on saturday was the “Writing About Python” BoF, which was cool, but in which I may have over-expressed several cough strong opinions.

The video of my second talk is here. This one, my lack of confidence is apparent, and I’m still fidgeting. I think part of my problem is I like to express with my hands, almost like a spastic air traffic control man and holding the mic/trying not to fidget made my body just sort of spaz. This time though, I rushed from my talk into Alex Martelli’s excellent “Abstractions as Leverage” talk. After that, more people found me and picked my brain.

I can’t stress how awesome Alex’s talk was — much of my hallway discussion time was spent discussing abstractions (especially around distributed systems) with him.

We’ll see how my call for some cohesion and a real “django for distributed systems” call in my second talk goes. Everyone I have spoken to likes the idea and had a lot of great feedback, but lord knows I don’t have the time to lead it up right now, maybe soon.

Late saturday, was the teach me web testing BoF which I ducked into, and then past that the Testing In Python BoF which provided me much in the way of fun, ideas, discussions and uh. Beer. Yeah. I can’t exactly remember what I said when Titus put me on the spot to discuss “what sucks about testing in python”. But I do remember having some great laughs. I think the heckling got recursive at one point, not sure though.

Sunday came, and with it, my sorry attempt to sit in the “Functional Testing Tools in Python” panel — which was thwarted by a 45 minute bloody nose. Apparently Titus talked a lot, so I don’t think I missed anything. End of main con.

Before I move into the sprints — I want to point out that I had fantastic hallway discussions with tons of people — people I look up to, and people I’ve never met before. I had excellent lunches with groups of people ranging from the highly experienced to someone who has learned python a week before. I can’t say enough about the simple fact that you get to relax, hang out and just talk with all of these people. Python is defined by it’s community — and ours is pretty awesome.

The sprints

Monday morning, I was off to the python core sprint. I immediately started hacking on multiprocessing bugs — I think all told, I managed to close around 12 or so bugs by the time thursday morning came around. The sprint was awesome simply due to the fact I could look across the room/table and ask any number of core developers questions. Martin Van Loewis came to my rescue/aid a few times, and being able to bounce ideas off of everyone is pure unbridled awesome.

Having dedicated face to face time to beat on these bugs and share information/debate things is immeasurable. I also got access to snakebite thanks to Trent, who is still working out the kinks. Alas, even having access did not make cross-platform support (mainly the BSDs) any easier for multiprocessing. I spent a good chunk of my time futzing with virtual machines so I could do further work — I ended up cutting that short because I wanted to fix things at the sprints: not fart around with tool chains.

Other than hacking until late at night through the sprints, there’s was quite a fun night where bourbon flowed, and many partook — much fun was had by all. There’s nothing quite like having drinks and making awful threading/process/compiler jokes.

Bug fixes/patch work for multiprocessing:

• multiprocessing/pipe_connection.c compiler warning (conn_poll)
• test_multiprocessing fails on systems with HAVE_SEM_OPEN=0
• Multiprocessing package build problem on Solaris 10
• with lock fails on multiprocessing
• Bus error when calling .poll() on a closed Connection from multiprocessing.Pipe()
• multiprocessing: SocketListener should use SO_REUSEADDR
• multiprocessing queues.py doesn’t include JoinableQueue in its __all__ list
• patches for multiprocessing module on NetBSD
• test_multiprocessing: test_listener_client flakiness
• multiprocessing: switch to autoconf detection of platform values: This one was a bear. All the props in the world go out to Martin Van Loewis for guiding me through getting this done.
• multiprocessing.Pipe terminates with ERROR_NO_SYSTEM_RESOURCES if large data is sent (win2000)
• implement initializer for multiprocessing.BaseManager.start()

In Summary

PyCon is simply getting better — I’m looking forward to Atlanta next year, if nothing more than for better weather. I got to talk to a lot of great people — Collin Winter, Thomas Wouters, Brett Cannon (be careful, he’s an odd one), Guido, Alex Martelli and many others. Having many hallway discussions with language users and encouraging them to get involved — PyCon is AwesomeCon — no where else do you get to hear so much great information, discussions and points of view.

All of the videos for this pycon are going up on pycon.blip.tv.

It was a great time, and I can’t thank everyone involved enough.

I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the author(s) of said tool(s)). I will also add additional bits and pieces as I learn and explore more./Note

Stackless python — here’s another big one on the pile — is much more than a library, or a framework which runs on CPython — Stackless is actually a modified version of the CPython interpreter. It’s much more than just a C-extension. Stackless is in use by various people and companies — most notably, it’s in use by CCP Games, makers of Eve Online (see this pycon presentation). In fact, CCP Games is a large part of why Stackless is still around today.

I say that intentionally — quoting the readme in the Stackless/ directory of the distribution(here):

In 2003, the fabulous PyPy porject was started, which is still
performing very well. I have implemented Stackless for PyPy
(with lots of support from Armin), and it is so incredibly
much nicer. No more fiddling with existing calling conventions,
no compromizes, everything that needs to be stackless also is.

Unfortuantely, PyPy is still not fast and complete enough to
take over. This means, my users are still whining for an update
all the time CPython gets an update.
And maintaining this code gets more and more a nightmare for
me, since I have the nice PyPy version, and I hate hacking this
clumsy C code again and again.

The original author, Christian Tismer largely moved onto PyPy, which is still largely in it infancy (although I read through bits of the code base frequently, it’s pretty), and further development has largely been stalled minus the improvements Richard M. Tew (CCP Games) and others have done. There’s still life in it.

Fundamentally, Stackless modifies the interpreter internals a bit to modify the way that the C call stack is manipulated/used as well as to add other other nice bits Stackless offers (call stack). Stackless simply doesn’t use the C call stack — all told, each microthread only has a few kilobytes of overhead, which is awesome.

It adds something called microthreads and does other patching to python-core. Normal OS/Posix threads require a fair amount of resources to create and run — in the case of Python, each thread has to get its own stack, this costs memory. With Stackless’ microthread support — you get “threads”, but threads which cost a significantly less, and potentially execute faster due to context switching improvements (no need to go from user->kernel->user and so on).

Point of Order: Before I continue, I want to clear up a common misconception I’ve heard — Stackless, does not in any way, remove the Global Interpreter Lock. No sir. It’s still there. Lurking. Waiting to steal your candy. Also, it still has a stack, so it’s not truly “stackless”.

So, microthreads are smaller and require less OS hand holding for context switching, and ultimately can (and are) scheduled by the interpreter, rather than the operating system.

install note for os/x users: You need to pass the “–enable-stacklessfewerregisters” to configure, otherwise, make pukes on you.

Stackless is a basic implementation of these — for a simple resource example usage example, I wrote a simple script which spawns 2,000 threads (sorry windows) and 2,000 tasklets. I watched the memory usage of both:

1
2
3
4
5
6
7
8
9
10

import time
import threading
def func():
    time.sleep(120)
 
threads = [threading.Thread(target=func) for i in range(2000)]
for i in threads:
    i.start()
for i in threads:
    i.join()

For the threaded script — the resident size was 42M and the virtual size was 1037. Versus the stackless version:

1
2
3
4
5
6
7
8
9
10
11
12

import time
import stackless
 
def func():
    for i in range(120):
        time.sleep(1)
        stackless.schedule()
 
for i in range(2000):
    stackless.tasklet(func)()
 
stackless.run()

Stackless had a resident size of 3416K and a virtual size of 22M — virtually microscopic versus the heavier thread version. Obviously, they are not line for line comparisons — the Stackless version, like other cooperative multitasking systems requires that each tasklet be a good citizen, and not block execution forever, instead rescheduling itself or otherwise yielding to allow for others to run. If a tasklet blocks on a socket, everyone blocks on that tasklet.

Someone asked me to track the linear growth of the threaded numbers vs. the tasklet numbers. Since I’m a sucker, I thought I’d take him up on it (OS/X 10.5, 4GB of ram, Core 2 Duo):

Threads:

Tasklets:

One note — the Stackless numbers should be low, but not this low (from my understanding, and review from others), anyone have any ideas?

There’s the numbers — lots of threads is going to consume lots of ram. With stackless, a given tasklet is only a few kilobytes in average size and therefore the memory footprint is small when you start raising the count. Additionally, note the two counts at the bottom of the tasklets table; you can’t spawn that many threads (depending on your OS and configuration) and even if you could, the memory footprint would be costly.

Now, in the age of cheap-ass-ram, where you can trick out a desktop or server with 16GB sticks, people might argue “so what” — but on machines where memory is constrained, such as smaller notebooks, embedded devices, or game consoles — this is a critical thing to take into consideration.

If you look at the stackless code, there is another big thing to realize; Stackless like other frameworks or systems which use an scheduler built into the interpreter gives you the benefit/task of scheduling when your tasklets/components/etc execute. This gives you more control, but more responsibility. Stackless offers both cooperative and preemptive scheduling, however the preemptive scheduling doesn’t feel right. more on scheduling here

So, we’ve determined that stackless tasklets are smaller, right? Pretty simple.

If you’ve read the other things I’ve written on Kamaelia/Twisted/etc, you’ll recognize the concepts within Stackless pretty quickly — a tasklet is a component, a thread of work and tasklets intercommunicate via channels. For example, here’s a little example of two tasklets communicating:

1
2
3
4
5
6
7
8
9
10
11
12

import stackless
 
def chicken(channel):
    channel.send('cluck')
 
def egg(channel):
    print channel.receive()
 
channel = stackless.channel()
stackless.tasklet(chicken)(channel)
stackless.tasklet(egg)(channel)
stackless.run()

Pretty easy, and a tiny amount of code. The concept of tasklets/microthreads isn’t a new one — in fact, it’s how Erlang gets it’s groove on — Erlang doesn’t use native OS threads, instead, it uses microthreads scheduled by the Erlang compiler. However, they are not directly comparable. Stackless isn’t running across cores — Erlang does, stackless, due to the GIL, has to obey the same rules as the rest of python-core. For more on “erlang v. stackless”, see this.

Oh, and you can share normal object via the channel too:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import stackless
 
def yes(channel):
    x = channel.receive()
    x.append('yes')
    channel.send(x)
 
def no(channel):
    x = channel.receive()
    x.append('no')
    channel.send(x)
 
channel = stackless.channel()
stackless.tasklet(yes)(channel)
stackless.tasklet(no)(channel)
channel.send([])
stackless.run()
print channel.receive()

Moving on, Stackless offers something else — the ability to pickle tasklets. This means you can pickle up a tasklet and send it over the wire to another machine and then unpickle it and continue running it — channels get pickled too. Locally, this means to can save it to disk, and then resume state easily.

You could use this to generate a tasklet which listened on a port for data on the local machine, and passed the data off the wire to the channel — when you pickled the channel or the tasklets with that channel in scope, and sent it over the wire, they would pick up listening on the same port number on the remote machine. You loose current sessions, yes, but you could also detect active sessions and handle those gracefully.

This is nice for say, a component you wanted to be able to easily send to other machines to help load balancing. In theory, you could auto-detect new servers being added to a cluster, and when that server came up into a “ready” state, send it the daemon it should handle — and the tasklets would pick up where they left off (minus the sessions).

Otherwise, pickling channels and tasklets could be used for a few things — you have to think of them in terms of coroutines (here) — you should be able to suspend processing state and then simply resume where you left off. If you’ve got python brains — picke-able generators. You could put them in a database; but the use of that escapes me at the moment.

Oh — and pickled tasklets and channels can be shared amongst different architectures too, as long as that architecture is running the same version of Stackless, and supports Stackless.

To continue on — if you wanted to add threads into the mix as well — Stackless tasklets can be run within Python threads, however those tasklets are local to that thread, and each thread gets it’s own scheduler. Your main application thread has it’s own scheduler, and so on.

Stackless’ tasklet/channel system is quite nice, however note that I’m not saying Stackless is the only way into this magical world — it’s not, especially with a plethora of coroutine/greenlet/etc packages for python today, and the continued work towards making generators more awesome. I’m just showing what Stackless can/could do.

The primitives within Stackless are nice — frankly, I’d like a light weight green thread implementation in python core on which we could build a nice Actor library, as well as support the lower memory footprint/etc. However, in order to use these primitives within Stackless — you’d find yourself building your own abstraction layer/framework (for example, concurrence) to really get a lot of mileage out of it. This is why people run twisted on top of it, CCP Games has the uthread library (which you can see here) and so on.

The cost of a deployment of Stackless can not be underestimated though — it’s got some magic assembler code within it, which isn’t the most portable of goods (versus OSes, compiler versions, compilers, etc). Some platforms simply aren’t supported due to this. Not to mention, it’s an entirely new interpreter, which has a cost much higher than that of an extension module.

A few people who I’ve been talking with asked me the simple question — “Why hasn’t any of this been pushed into python-core”. Well, in short — it was never really proposed (by Christian), and the changes within Stackless — the last serious discussion was from 2007 (see this).

With Stackless, it’s difficult — I think there is a perceived complexity about the code and then there is real complexity. I suspect both of these are high in the case of Stackless due to the nature of the problem it is trying to solve; namely bolting a feature like this onto an interpreter not meant for it. I think that due to this, and due to Christian and others moving onto the greener pastures of PyPy — inclusion into core simply won’t happen.

Resources:

• http://www.onlamp.com/pub/a/python/2002/02/14/pythonnews.html
• http://www.python.org/dev/peps/pep-0219/
• http://www.python.org/dev/peps/pep-0255/
• http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/
• http://www.ibm.com/developerworks/library/l-pythrd.html
• http://mail.python.org/pipermail/python-dev/2007-February/071037.html
• http://members.verizon.net/olsongt/stackless/why_stackless.html
• http://code.google.com/p/stacklessexamples/wiki/StacklessExamples
• http://islab.org/stackless/

PyCon 09 is turning out to be one of the ones I am most excited about in some time — barring the fact they let me actually stand up and speak about something, there’s a ton of other excellent and exciting things going on.

I will be doing two talks — “Introduction to Multiprocessing in Python” on Friday, at 3:20 PM, and “Concurrency and Distributed Computing with Python Today” at 3:20 PM Saturday.

The former talk is easy, given it will be focused on introducing the multiprocessing module to the masses, and taking questions about it (be gentle, I just work here). The latter is a much bigger beast. I’ve been blogging about my research in my pycon 2009 category, and I still have a pile of things to keep adding to that. The talk will attempt to differentiate concurrency from distributed systems, and show the various toolkits/frameworks/etc in the ecosystem today, to help you build both types of systems.

Given both talks are 45 minutes in length, I will be publishing my talk notes (my slides are not going to be heavy weight) and other information here.

In addition to me speaking, which may or may not be exciting, there’s one helluva ton of other talks which simply look awesome.

My schedule looks like this:

• Thursday: Python Language Summit, where I will endeavor to be smart.
• Friday: How to give a python talk
• Friday: Using Windmill
• Friday: Introduction to Python Profiling or How Python is Developed, to harass Brett.
• Friday: Panel — Python VMs
• Friday: Building an Automated QA infrastructure using Open Source Tools
• Friday: Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications
• Friday: My Talk (I figure I should go to it)
• Friday: A Whirlwind Excursion through Writing a C Extension or Challenges and Opportunities for Python
• Thursday: Plugins and monkeypatching: increasing flexibility, dealing with inflexibility
• Saturday: The (lack of) design patterns in Python or the Pinax talk
• Saturday: Class Decorators: Radically Simple
• Saturday: Panel: Object Relational Mappers: Philosophies and Design Decisions.
• Saturday: Drop ACID and think about data (I wonder if we can take this literally, if he has scary slides though, we all might trip balls)
• Saturday: My Talk, which is up against Bruce Eckel and Raymond H — I expect no one to show up.
• Sunday: Panel: Functional Testing Tools in Python
• Sunday: Designing a web framework: Django’s design decisions

This doesn’t even cover the open space discussions which I might attend — including the “Writing About Python” one Doug Hellmann is putting together as well as the “Teach Me Web Testing” one by Steve Holden.

After the main conference, I’ll be sticking around until Thursday morning for the sprints, at which point I should be sufficiently burned out on python stuff, I will fully convert over to being a full time burger flipper.

Right now there are 620 registered people who made their attendance public I don’t know how many there are in total.

Hope to see you there!

I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the author(s) of said tool(s)). I will also add additional bits and pieces as I learn and explore more. Additionally, thanks to glyph for giving me a hell of a lot of feedback./Note

Twisted is the 800 lbs gorilla of the “concurrency” frameworks. It’s been around for awhile, has a large following — it’s used by everyone from Apple (iCal server) to Buildbot Buildbot. It has a literal ton of sub projects and other “semi attached appendages”.

Twisted can be daunting for almost everyone — while it is conceptually simple, the docs and examples could be more approachable. People look at Twisted as an all-or-nothing bet when considering it for their applications, which to an extent, it is.

But if I am going to do a concurrency/distributed systems talk — I can’t ignore one of the most widely used and original frameworks in this space.

So, as always — these are my semi-rough notes diving into Twisted-core. Like Kamaelia, I am going to side step the asteroid ring which surrounds Twisted (or tentacles… I can’t decide which to use) and delve into the core.

Moving along — Twisted is based around asynchronous programming — a model for adding a great deal of concurrency to your application via non blocking calls or isolating blocking calls “elsewhere”. This is largely the same approach that GUI toolkits use wherein a given event is assigned something to run when an “event” occurs, such as a button click or data is available. Glyph sent me a very simple side-by-side:

1

    reactor.listenTCP(8080, someFactory)

1

    button.connect('clicked', someCallback)

This shows the essential aspect of asynchronus/event-driven systems — you tell x that when y occurs, call z. It really is conceptually simple. The main loop of the application simply focuses on constructing these relationships, and executing the callbacks when the event occurs.

Twisted’s focus is on network-based applications — these mesh well with the idea of non-blocking I/O, where you have to fundamentally chunk your work up into small pieces which take a very short time to execute. Networked apps spend most of their time waiting for data to come in over the wire. Twisted also has faculties to isolate CPU intensive and/or blocking calls within threads or processes.

I’m going to focus on two of the core components — Deferreds and the Reactor, this should help illustrate what the core paradigm is.

Deferreds are a core component of Twisted — a deferred in the simplest terms is an object that when created, represents some Thing which will eventually return Something or Error — a placeholder for something in the future. The way you handle Something or Error is you tell the deferred that if it gets Something, it should call consume_function, otherwise — if it gets an Error, it should call error_function. Easy!

Take the below for example — read_mail and error_function are what’s known as callbacks — a function which is called by something else when something occurs (I hate me for writing that).

Callbacks are really simple, as illustrated here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import os, sys
def read_mail(mailitems):
    print mailitems
    sys.exit()
 
def read_error(error):
    raise Exception('error: %s' % error)
 
def wait_for_mail(callback, errback):
    while True:
        try:
            mail = os.path.isfile('mail')
            if mail:
                callback(open('mail','r').readlines())
            else:
                pass
        except Exception, e:
            errback(e)
 
wait_for_mail(read_mail, read_error)

Sidenote: Callbacks are stupid easy with python. I love them. Heck, you can pickle a callback and shoot it over the wire to another machine. Callbacks are cool. You should watch Alex Martelli’s callback talk on youtube.

In any case, you tell a deferred (the object which represents a promise of something) what to do when data is returned — you do this by generating a deferred, and then adding callbacks onto it — note that the function wait_for_mail needs to return a deferred. In this toy example, I want to just look for a “mail file” on disk, and then if it exists, return a string to the callback:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import os
from twisted.internet import reactor, defer
 
def read_mail(mailitems):
    print mailitems
    reactor.stop()
 
def wait_for_mail(d=None):
    if not d:
        d = defer.Deferred()
    if not os.path.isfile('mail'):
        reactor.callLater(1, wait_for_mail, d)
    else:
        d.callback(open('mail','r').readlines())
    return d
 
deferred = wait_for_mail()
deferred.addCallback(read_mail)
reactor.callLater(60, reactor.stop)
reactor.run()

I wanted to keep this as simple as possible, as it illustrates some things that fundamentally tripped me up at first.

First typically, if you were to solve a problem — say, polling a mailbox, you might do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import os, sys, threading, time
def read_mail(mailitems):
    print mailitems
 
def wait_for_mail(reader):
    while not os.path.isfile('mail'):
        time.sleep(.1)
 
    reader(open('mail','r').readlines())
 
t = threading.Thread(target=wait_for_mail, args=(read_mail,))
t.start()
t.join()
sys.exit()

Or some other pattern of spawning a thread and then waiting for that thread to chuck the data back to you. The thread gets out of your way, and doesn’t force the main part of the program to block, sort of. In fact, with Twisted, the main part of your application is required not to block.

As a point of order, you could do the same thing with Twisted like this:

1
2
3

...
d = deferToThread(wait_for_mail)
...

Threads are the common way of pushing blocking work off to the side — most of us have had to do it at one point or another. In the Twisted world, this has to be turned on it’s head a bit.

In Twisted, you have to split your problem into small, individual functions/methods — ideally, you isolate the really blocking part (say, waiting for a file to appear) in it’s very own function. You make the non blocking parts — say checking initial existence and constructing the deferred object run as quickly as possible, and then return the deferred — a promise of data to come via the slow method. The slow part in the threading example is the time.sleep().

You then schedule that blocking call to run, it shouldn’t block, but rather it should poll for data or changes in it’s buffer(s) and either reschedule itself to run if there is no data, or return the data if there is. This event is time based — but the same applies to adding a callback to a non-time-based item, such as setting up something which listen on a socket for data.

The fact you schedule work within the reactor tripped me up at first. I was thinking in blocking terms though (Twisted has a faculty for passing blocking work off to threads via the deferToThread call) — the function kept wanting to block instead of polling, or looking for state change. Everything needs to be scheduled in one way or another.

Glyph wisely pointed out that this is a common issue with people rethinking concurrency in terms of “discrete events”. For example, most people are content to think about concurrency in terms of workers “who are off doing things”. In theory, those workers are “always doing something” — in reality, the operating system is simply suspending your worker(s) until an interrupt (i.e.) discreet event occurs, which causes the worker(s)/app unblock.

Glyph suggest the following as a good example using generator syntax:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from twisted.internet.defer import inlineCallbacks, returnValue
from twisted.internet.task import deferLater
from twisted.internet import reactor
 
def deferredSleep(howLong):
    return deferLater(reactor, howLong, lambda : None)
 
@inlineCallbacks
def wait_for_mail():
    while True:
        if os.path.isfile('mail'):
            returnValue(open('mail','r').readlines())
        yield deferredSleep(1.0)
 
@inlineCallbacks
def check_mail():
    mail = yield wait_for_mail()
    print 'got mail', mail

The example he provided is interesting — it has no “scheduling” involved, and instead it uses something I didn’t see originally — deferLater and inlineCallback, inlineCallback accepts a function as an argument, that function can yield a deffered or call returnValue, essentially anywhere where you would normally block, you simply yield. In this case, we simply call deferLater if the file doesn’t exist, which tells the reactor to re-run this some time in the future.

Before I move off the basic view of deferreds — there’s something to note, deferreds can accept a chain of callbacks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import os
from twisted.internet import reactor, defer
 
def read_mail(mailitems):
    print mailitems
    return "this %s is junk" % mailitems
 
def shred_mail(mailitems):
    print 'buzzzzz: %s' % mailitems
    reactor.stop()
 
def wait_for_mail(d=None):
    if not d:
        d = defer.Deferred()
    if not os.path.isfile('mail'):
        reactor.callLater(1, wait_for_mail, d)
    else:
        d.callback('letter')
    return d
 
deferred = wait_for_mail()
deferred.addCallback(read_mail)
deferred.addCallback(shred_mail)
reactor.callLater(60, reactor.stop)
reactor.run()

And the output:

letter
buzzzzz: this letter is junk

This allows each callback chained onto a deferred to alter the data in some form — the modified data is passed to the next callback in the chain. It’s not really magic. It’s simply a series of functions to call when an event occurs.

One thing to keep in mind when thinking about Twisted — Twisted is single threaded. Ok, not really — sort of. Glyph called me out on this, and rightly so. In reality all I/O in Twisted is single-threaded, most Twisted APIs are not thread safe and deferred callback chains execute in a single thread… But — Twisted does support both threads and processes. There is no reason why you can’t use a library which uses threads in your twisted application for example. You can use threads with twisted, so don’t worry too much about that.

On the other hand, if your entire application is a thread-spawnfest, you might want to reconsider — the design of your app, that is. /flamebait

All code executes in the main thread of a single python process, which is why when discussing deferreds and blocking calls, it is so important to break your problem down into the smallest steps possible and isolate/rework blocking code into deferred actions.

Onto the reactor then.

The reactor is the event loop mechanism for Twisted. It takes care of executing all of the various timed actions and the execution of the callback/errback stack. Timed actions can be deferreds, etc. Deferreds are simply objects executed by the Reactor.

You’ll notice in the example above, we didn’t create an instance of the reactor, instead we just imported it. If you look at Twisted.internet.reactor — you’ll see this removes any previous instances of reactor in sys.modules and then calls install() on the target reactor. This means the reactor is a singleton, all future imports/calls will always refer to this reactor.

Now, the documentation in internet.reactor mentions that new application code should pass around an instance of a reactor — this moves away from the reactor-as-a-singleton (the simple behavior) and into something a bit more interesting. Glyph pointed out the obvious usefulness for this — testability, you can pass in a reactor rather than grabbing it from the global, you have more control.

You can also use this to group a series of actions both timed and otherwise, connections/etc within a given reactor, and then create a meta-reactor to control the multiple reactors. Reactors, all the way down.

See, Twisted has multiple types of reactors — there are reactors based on select (the default), GTK, Cocoa (PyObjc), etc. Each reactor manages the scheduling and execution of the tasks added to it in it’s on unique style, but implement a common interface. Application code should not care what reactor is running — the reactor is an abstraction above other, some operating system optimized polling/looping mechanisms. For example, GTK+ looping and polling, select on Linux/etc.

Quoting Glyph:

The idea of all this is that Twisted code is at the top of the food chain.
GTK+ networking code can only run in GTK+ programs, kqueue networking code can
only run on FreeBSD machines, but Twisted networking code can run anywhere it
can get its hands on something that looks even vaguely like select(). If you
want a Windows desktop program and a UNIX server program to run the same
networking code, but have radically different event APIs under the covers for
good performance, Twisted has you covered.

You can install a given reactor by doing the from twisted.xxx.reactor_name and calling the install() method (which is all twisted.internet.reactor does). The preferred method is to use the “–reactor” argument to the twistd/trial tools.

Twisted is fundamentally a networking stack: it’s built to solve non computationally intensive tasks that require a lot of waiting and polling for data: networking fits perfectly into this. With additions — it can also serve as a platform for computationally intensive/distributed applications. For example, the ampoule add on.

In the networking sense: Twisted is perfect as long as you can deconstruct and rethink the way you solve day to day problems, remove or rethink blocking code, switch your application model to something event/message driven.

Admittedly; Twisted isn’t for me (right now) — but with time, it could be, and it could work great for your application today. It has libraries for just about any protocol you could possibly ask for. It’s code base is huge and actually has some pretty cool code inside of it. The catch is — you don’t port an application to Twisted: You write a Twisted application in most cases.

However; it can also be hard to approach — both within the code, and the documentation. You see questions pop up all the time “I think Twisted does this” — and while it probably does it could take you awhile just to grok what it is to be Twisted.

For example — dumb down the examples. While the finger tutorial is “the introduction” starting down on a level where you take a normal, single threaded application (perhaps using generators), port it to threads, point out the issues there, and then port it to twisted, using as little “twisted magic” as possible.

This isn’t to say it’s impossible to grok; but helping walk people through learning how to begin to think asynchronously, rather than explaining the esoteric or the One True way of doing something, consider the positive feedback that Steve Holden’s “Teach Me Twisted” got (summary here). Assume most people can’t spell asynchronous, and build it up.

The Django documentation is probably one of my favorite examples of this — it starts very simple and very approachable and walks the user through everything “from the beginning”.

Related Links

• http://stackoverflow.com/questions/80617/asychronous-programming-in-python-twisted/81456#81456
• http://www.usenix.org/events/usenix03/tech/freenix03/full_papers/lefkowitz/lefkowitz_html/index.html
• http://mumak.net/stuff/Twisted-intro.html
• http://sluggo.scrapping.cc/python/Twisted_finger_gentle.txt
• http://Twistedmatrix.com/projects/core/documentation/howto/defer.html
• http://Twistedmatrix.com/projects/core/documentation/howto/gendefer.html
• http://Twistedmatrix.com/projects/core/documentation/howto/async.html
• http://Twistedmatrix.com/projects/core/documentation/howto/reactor-basics.html
• http://www.nightmare.com/pythonwin/async_sockets.html
• http://enthusiasm.cozy.org/archives/2009/02/twisted-nevermind

Let me state this: these are slightly cleaned up versions of my notes as I am learning these modules — some of them have higher learning curves for me than others.

Circuits is, well — an event based “framework” (again, note the small f) based around the concept of Components (big C!) consuming/reacting and in turn generating events — all asynchronously.

James’ goals seem pretty simple — build something with no external dependencies, that’s compact (I should say, the core.py is < 500 lines) and that makes it easy to build scalable messaging based (event based) systems. Did he succeed — don’t know! Let’s dig in.

Note:Given his site is/was down at the time of this writing, I simply cloned the hg repo (here) and used that for all the code and documentation, which is why I am going to sparse on direct links to the documentation. Also note, that pulling from tip introduces dependencies on python 2.6.

Starting with the quickstart — always a good place to jump to, James outlines the very basics — as in, the very, very basics via a simple code example, which I have paraphrased, renamed the objects of and… you get the picture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

from circuits import listener, Event, Component, Manager
 
class AnEvent(Event):
    "AnEvent Event"
 
class TheComponent(Component):
 
    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"
 
def main():
    manager = Manager()
 
    thecomponent = TheComponent()
 
    manager += thecomponent
 
    for i in range(10):
        manager.push(AnEvent(), "anevent")
 
    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

And of course, if you run this — you see “hello, I got an event” print 10 times. Simple! But here there be magic, so let’s break it down. First, the events.

Here, we subclass and generate a new event:

1
2

class AnEvent(Event):
    "AnEvent Event"

If you look in core.py, you’ll find that Event is a container — there’s some magic here at first glance — but really all this does (see __new__) is construct a new object of the passed in class, and chain any arguments into attributes. The rest is just accessing and comparison — for example:

1
2
3
4
5
6
7
8

>>> from circuits import Event          
>>> class x(Event):
... 	"an x event"
... 
>>> y = x(1,2,3,4,arg_one='foo',arg_two='bar')
>>> print y
<x/ (1, 2, 3, 4, arg_two=bar, arg_one=foo)>
>>>

This means, that you can pass in any number of positional arguments and keyword arguments, and they’ll be packed into your event. More on this later.

Moving on, the next thing we define is TheComponent:

1
2
3
4
5

class TheComponent(Component):
 
    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The parent class of this object is a little bit more magical. James doesn’t seem shy of the metaprogramming:

1
2
3

class Component(BaseComponent):
 
    __metaclass__ = HandlersType

Nuts. Right above it though is the definition of BaseComponent which is slightly simpler but reveals something interesting — each component can be passed in a channel, and the default is None (more on channels in a bit). Additionally, we see BaseComponent is a subclass of a Manager. Components are managers, and managers are components.

Finkle is einhorn, einhorn is Finkle!

The metaclass does magical object construction stuff and method management/etc. Dragons.

So, the superclass has two addition methods — register() and unregister() which control the component’s registration with the manager (more later). Otherwise, it doesn’t have a lot else, except for the on_anevent() method we’ve added.

And made fancy with a decorator (mandatory pieces of flair)!

This code bit:

1
2
3

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

Essentially means “this method reacts to this event” — so, let’s look at the listener decorator a bit. The decorator is in core.py, and the docstring explains a lot more about the various arguments the decorator can take, but not very clearly. I guess you had to be there.

The one thing you notice though, is that the decorator adds some method attributes to your method — things like f.type, f.target, etc. And then something called f.br, the use of which isn’t clear to me right now. You can however have a single handler listen for multiple events:

1
2
3
4
5

class TheComponent(Component):
 
    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"

note: James explained the .br attribute:

handler.br is used in the BaseComponent’s send(…), and iter(…). It is used to determine which “branch” to follow. The args and kwargs of an Event are intelligently applied to an event handler depending on the signature of the event handler. (It took some time to get right — but basically it means you can’t go wrong!)

In short — we now have a registered listener for an event named “anevent” — when an event is pushed (more on this later). More recently James altered it so that if you define a component, any method which doesn’t start with a _ in the name will explicitly become an event handler of “listener” type and listens on a channel which is based off the method name.

We can refactor the original code like this:

1
2
3
4

class TheComponent(Component):
 
    def anevent(self):
        print "hello, I got an event"

And it works just as well — which is a nice improvement. Here’s the note from James on this:

Basically, the new HandlersType (metclass) means that every method defined in a sub-classes Component that have not been previously defined as event handlers with the @listener decorator or do not start with a _, are automagically turned into event handlers listening on a channel that is the name of the method.

The use of the @listener decorator is still required for:

• Defining filters
• Defining event handlers listening on multiple channels.
• Defining event handlers listening on foreign targets.

Next up is the main function — here we instantiate a new Manager object, which if you look at the code of this, you’ll notice right off the bat that there’s a startling number of __ method overloading. Which explains this line:

1

    manager += thecomponent

This is handled by the manager’s __iadd__ method:

1
2
3
4
5

    def __iadd__(self, y):
        y.register(self.manager)
        if hasattr(y, "registered"):
            y.registered()
        return self

This means that when you append it onto the manager (something more explicit would be nice — brevity and terseness be damned) it calls the register function on the component. Reading through that method is actually quite telling as it introspects the current object (a component) and extracts the callable methods and then pulls out the handlers/channels for a given event and calls .add on the manager. A given component may have multiple listeners for a given event inside of a component:

1
2
3
4
5
6
7
8

class TheComponent(Component):
 
    @listener("foobar", "anevent")
    def on_anevent(self):
        print "i listen for foobar and anevent"
 
    def anevent(self):
        print "hello, I got an event"

These handlers are all registered with the manager. Now, something I want to clear up is that when you declare:

1
2
3

    @listener("anevent")
    def on_anevent(self):
        print "hello, I got an event"

The “anevent” defines the channel that listener listens on. It’s easy to say “listens for events” — but really what this is is a subscription to a channel. This means you can define a listener with, well — no event:

1
2
3

    @listener()
    def all(self):
        print "i listen for everything"

It’s like having all the premium channels.

Back to the manager though (the above was a light bulb going off in my head) — next we see that we “push” events into the manager. This is actually pretty simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

    def push(self, event, channel, target=None):
        """E.push(event, channel, target=None) -> None
 
        Push the given event onto the given channel.
        This will queue the event up to be processed later
        by flushEvents. If target is given, the event will
        be queued for processing by the component given by
        target.
        """
 
        if self.manager == self:
            self._queue.append((event, channel, target))
        else:
            self.manager.push(event, channel, target)

This means we can push events into the manager, and target them at a specific component. The definition of an event in the case of a listener is actually the creation of a channel.

The push method exposes something else — it seems possible to create a manager which is actually a pointer to another manager (a proxy) alas, I don’t see how to do that. Hooray!

So, we can create new events and push them into the manager which pipes them off to the channels who have assigned listeners. Easy! We flush the manager’s queue so all events are pushed to the components, and we’re done.

So what can we use these basics for? Easy — building components which stack on top of each other which contain other components which subscribe to a given event. In the examples directory, James has done an excellent job showcasing a lot of problems which he solves using the core circuits library. In fact, the examples explain a lot more about how things work than the core code itself.

One of the questions I had in working though all of this, is what happens if you define a general event — can the handler gain access to arguments within the event? Does it need to? Is it simply sufficient for a listener on a given channel to know the name/type of an event and react to it?

The answer is, well, yesandno. Let’s say we do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

from circuits import listener, Event, Component, Manager
 
class AnEvent(Event):
    "AnEvent Event"
 
class TheComponent(Component):
 
    def anevent(self, *args, **kwargs):
        print args, kwargs
 
def main():
    manager = Manager()
 
    thecomponent = TheComponent()
 
    manager += thecomponent
 
    for i in range(10):
        manager.push(AnEvent(1,2,3, no='yes', yes=False), "anevent")
 
    while True:
        try:
            manager.flush()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

You would see:

1
2
3
4
5
6
7
8
9
10

(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}
(1, 2, 3) {'yes': False, 'no': 'yes'}

Awesome. The arguments the event is created with are passed directly into the listener for that, so as long as you have the right signature on the method, you should be golden.

Which leaves us with a basic question — if messages have to be pushed into the manager on a given channel, how do we build something which is an event generator? In other words — how do we make something which “listens” and then generates the matching events.

To know that — we look in the lib/ directory (which the examples make prodigious use of) and we’ll focus on io.py which listens on stdin:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

from circuits import listener, Event, Component, Manager
from circuits.lib.io import Stdin
 
class DogBot(Stdin):
 
    def read(self, *args, **kwargs):
        print args, kwargs
 
def main():
    manager = Manager()
    dog = DogBot()
    manager += dog
 
    while True:
        try:
            manager.flush()
            dog.poll()
        except KeyboardInterrupt:
            break
 
if __name__ == "__main__":
    main()

All this does is create a new component which listens for the read event that Stdin generates and then prints off what we see coming in off the command line:

1
2
3

thumper:circuits jesse$ python2.6 dog.py 
a line is one argument
('a line is one argument\n',) {}

That means the read event needs to parse the passed in line and then (re)issue and event for some other listener to manage. Let’s define some addition events:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

class Sit(Event):
    """ Sit event, no args """
 
class Say(Event):
    """ Speak event
     args: what to say
    """
 
class DogBot(Stdin):
 
    def read(self, *args, **kwargs):
        command = args[0].strip('\n').split()
        if not command:
            return
        if command[0] == 'sit':
            self.push(Sit(), 'sit', self.channel)
        elif command[0] == 'say':
            self.push(Say(' '.join(command[1:])), 'say', self.channel)
 
    def sit(self):
        print 'i am now sitting'
 
    def say(self, words):
        print '<dog>%s</dog>' % words

Rudimentary — but you get the idea. For networking, you’d need to bind the socket and then read off/generate the events — this is largely covered by the sockets.py module in lib, as well as irc, webserver and smtp listeners.

All in all — it’s pretty simple to construct components, it could be made a bit easier with less metaprogramming and more, well, methods but a great deal more documentation could help too. I’m not too fond of too much metaprogramming — I tend to think it makes code rather unapproachable in general.

The (event/channel)/subscription model used here is nice as well — you can easily create your own little asynchronous network daemon or something as simple as what I did above very quickly (once you know what’s going on). It is obviously still evolving — James is putting a lot of work into it. If you’re looking for a compact little library, this would be good to check out.

I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the author(s) of said tool(s)). I will also add additional bits and pieces as I learn and explore more. Code and examples will be checked into my pycon 2009 bitbucket site here/Note

For awhile now, I’ve been meaning to dig into Kamaelia but was largely put off by what I tend to call the “twisted effect”. What this means is that when I go looking for libraries and small components, I go looking for a library — not a “solution”. I also worry about the “once you go in, you must follow this paradigm” effect. I’m not going to say that these feeling continue to be founded, or are completely rational — after all, I am digging into it, yes? It’s the thought that once you adopt the “one true way of doing things” you’re trapped in that solution/framework “forever” — ironically, I love Django for it’s “conceptual integrity” and full-stack approach. No, I don’t understand me either.

Also, as time has progressed, I have found that part of me yearns for a clean and simple to use “framework” (note the small “f”) that would help build out a large system without introducing a lot of complexity. Ideally, that framework would allow me to swap components in and out — think of web frameworks like django and turbogears — in this case, instead of using stock localized IPC, I might want to swap in a simple messaging protocol (Pyro, XMPP, etc).

It’s also a matter of marketing and approachability — things have improved on both websites mind you. Looking at Kamaelia’s website though, I don’t find it approachable, as it’s not immediately clear what the core idea is, or what the difference between Axon (the core) and Kamaelia (the project) is. For example, if I had one critique, I would say that Axon should become it’s own “project”/library in and of itself, and almost have it’s own website. It would be like ripping Twisted’s reactor out and making it a completely separate library.

Kamaelia, like Twisted, is based on a “simple” core — in this case, it’s the Axon library which has some very simple goals and paradigms it seeks to fulfill. To quote the Axon page:

Axon is a component concurrency framework. With it you can create software “components” that can run concurrently with each other. Components have “inboxes” and “outboxes” through with they communicate with other components.

A component may send a message to one of its outboxes. If a linkage has been created from that outbox to another component’s inbox; then that message will arrive in the inbox of the other component. In this way, components can send and receive data — allowing you to create systems by linking many components together.

Each component is a microprocess — rather like a thread of execution. A scheduler takes care of making sure all microprocesses (and therefore all components) get regularly executed. It also looks after putting microprocesses to sleep (when they ask to be) and waking them up (for example, when something arrives in one of their inboxes).

This by itself is the shining gem of the Kamaelia ecosystem — everything else is applications or additional utilities built on this simple core. This is where the website confusion comes in — where does “solutions built with axon (e.g. kamaelia)” end and Axon begin? The core design (of Axon) is very simple though: build a component which communicates via message passing.

Message passing is a relatively simple concept. Component A generates some work, and then sends it to Component (Not A). Messages are handled by the receiver and results can be passed (via a message) to someone else.

Very, very simple. You can add on little factoids about the fact that messages sent and received are handled in asynchronous fashion, messages can be sent locally — or across a wide network, etc — but largely those are component implementation details.

Which gets us back to Axon.

Since I’m interested in the core — and not video transcoders — I hit up the MiniAxon tutorial here and worked through it — even then, I don’t really think it did Axon complete justice. I then jumped into the “How to write new components” article by Michael.

The second tutorial, in my humble opinion, should be the first article users are directed to, while it has some polishing issues, I found it to really explain what the fruit was going on — and what Axon is.

Reading through both of these, you begin to realize that Axon is built on the core concept of Python generators and yielding control to a scheduler. For example:

1
2
3
4
5
6
7
8

def sender():
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
   sock.bind((ANY,SENDERPORT))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 255)
   while 1:
      time.sleep(0.5)
      sock.sendto("Hello World", (MCAST_ADDR,MCAST_PORT) );
      yield 1

If you’re familiar with python generators, you know that what this function does is send a message to a socket, and then hand control back to the main program. It will do so forever until the controller exits.

This paradigm is key to Axon: components send and receive messages via mailboxes (by far, one of the best descriptions/abstractions I’ve seen for this) — the components do the work sent/generated and then put it in the right outbox, and then yield control, courtesy of Enhanced Generators (see PEP 342)

Yes, coroutines/greenlets/tasklets — stop bothering me.

In the tutorial I linked above, Michael takes the very simple network script and ports it to Axon. Here’s my simple experiment that drops the networking code and cuts to the mailbox system. In this case, all I want to do is send and receive the lyrics to the meow mix commercial, forever:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

import Axon
 
LYRICS="I want chicken I want liver Meow Mix Meow Mix Please Deliver."
 
class Producer(Axon.Component.component):
    def main(self):
        while 1:
            self.send(LYRICS, "outbox")
            yield 1
 
class Sender(Axon.Component.component):
    def __init__(self):
        self.__super.__init__()
 
    def main(self):
        while 1:
            if self.dataReady("inbox"):
                message = self.recv()
                self.send(message, "outbox")
            yield 1
 
class Receiver(Axon.Component.component):
    def __init__(self):
        self.__super.__init__()
 
    def main(self):
        while 1:
            message = self.recv()
            print message
            yield 1
 
def tests(): 
    from Axon.Scheduler import scheduler 
 
    class testComponent(Axon.Component.component): 
        def main(self): 
            producer= Producer()
            sender = Sender()
            receiver = Receiver() 
 
            self.link((producer, "outbox"), (sender, "inbox"))
            self.link((sender, "outbox"), (receiver, "inbox"))
            self.addChildren(producer, sender, receiver)
            yield Axon.Ipc.newComponent(*(self.children)) 
            while 1: 
                self.pause() 
                yield 1
 
    harness = testComponent() 
    harness.activate() 
    scheduler.run.runThreads(slowmo=0.1) 
 
if __name__=="__main__": 
    tests()

Now, this uses knowledge from both tutorials, and the Axon.Component documentation. The component documentation can be hard to find, and not so easy to navigate too.

If we break one of the classes/components down and look at the “magic” provided by the component subclass, it gets clearer — in this case, I’ve “added” back in the methods from the component superclass, minus the doc strings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class Sender(Axon.Component.component):
    # First, subclass the Axon Component class, this provides us with the
    # basic inbox/outbox static members that look like this:
    Inboxes = { "inbox"   : "Send the FOO objects to here",
                "control" : "NOT USED",
              }
    Outboxes = { "outbox" : "Emits BAA objects from here",
                 "signal" : "NOT USED",
               }
 
    def __init__(self):
        self.__super.__init__()
 
   def recv(self, boxname="inbox"):
      # returns the first piece of data in the requested inbox.
 
      return self.inboxes[boxname].pop(0)
 
   def send(self, message, boxname="outbox"):
      # appends message to the requested outbox.
 
      self.outboxes[boxname].append(message)
 
   def dataReady(self,boxname="inbox"):
      # Returns true if data is available in the requested inbox.
 
      return self.inboxes[boxname].local_len()
 
    def main(self):
        while 1:
            if self.dataReady("inbox"):
                message = self.recv()
                self.send(message, "outbox")
            yield 1

I think this makes it abundantly clear what’s happening with the method calls on this class. Now, there’s the additional magic of the new tests method — in which we defined a new component, which was actually a component containing and linking the pipelines (connections between mailboxes) between the other components.

Now, the Axon.Ipc.* docs aren’t the most helpful — in our case, we called:

1
2
3
4

    self.link((producer, "outbox"), (sender, "inbox"))
    self.link((sender, "outbox"), (receiver, "inbox"))
    self.addChildren(producer, sender, receiver)
    yield Axon.Ipc.newComponent(*(self.children))

Within the main method of a component. We need to look at the link method on the superclass:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

   def link(self, source,sink,*optionalargs, **kwoptionalargs):
      """\
      Creates a linkage from one inbox/outbox to another.
 
      -- source  - a tuple (component, boxname) of where the link should start  
                   from
      -- sink    - a tuple (component, boxname) of where the link should go to
 
      Other optional arguments:
 
      - passthrough=0  - (the default) link goes from an outbox to an inbox
      - passthrough=1  - the link goes from an inbox to another inbox
      - passthrough=2  - the link goes from an outbox to another outbox
 
      See Axon.Postoffice.postoffice.link() for more information.
      """
      return self.postoffice.link(source, sink, *optionalargs, \
                                  **kwoptionalargs)

This leads us to the Postoffice class which actually constructs and tracks the links between the components. Here there be dragons.

So, addChildren just registers all of the passed in component instances as children of the newly constructed components (components, all the way down), and then we yield ourself — if you added a ‘print harness’ you’d see:

1
2

Component __main__.testComponent_5 [ inboxes : {'control': [], 'inbox': []}
outboxes : {'outbox': <>, 'signal': <>}

This means we’re getting a component back, and then calling .activate on it — .activate is actually a method on the Axon.Microprocess.microprocess class. In our case, it (it being activate) is simply registering our test component (which contains all of the children) with the default scheduler.

At which point, we call scheduler.run.runThreads jazz hands.

I dived into some of the internals here, I ended up having to supplement the documentation with pouring through the code — but I personally think it helps clear things up to remove some of the magic and show what is actually occurring. Most of the time, it seems you simply won’t care — and instead you’d just make and register your happy component and be on your way.

To somewhat summarize what we’re lookin at — a component that subclasses the default Axon.Component uses generators to yield control back and forth, passing messages/work to and from each other via the very clever mailbox/postoffice metaphor.

Now, the interesting thing, once you start digging through things is that your component isn’t a really a Thread — if you wanted to make sure each component was in its own thread, you might instead subclass Axon.ThreadedComponent.

Subclassing this new class looks mighty close to what we did before but instead Uses threads and queues for the message passing. Instead of yield, you just run, and the recv/send methods are backed by queues. Ahh, delicious non judgmental queues.

In any case, Kamaelia — via Axon, is a very nice abstraction on top of a very simple concept — message passing for concurrency. The fact that you can quickly build up a series of components which pass work back and forth via some sort of communications system and not have to worry about the underlying nuances/organization is quite nice.

One of the things Michael Sparks and I have talked about is adding some level of multiprocessing support for Kamaelia — this would actually be insanely easy if I used Axon.ThreadedComponent as the template, but instead used multiprocessing.Queue and multiprocessing.Process as the back end.

Kamaelia itself, is really a series of example components/applications which build on a core (Axon), but you do not need Kamaelia to use Axon effectively. In fact, just on a whim, I decided to whip up a dirty http load tool in Kamaelia. You can see it here.

It’s really a hack job — all I did was build off the meowmix demo, swapped in the threaded component and hacked it around a bit. One thing I’d like to know is how to pass in a dynamic number of clients so that I could create the outboxes dynamically in the producer — there wasn’t anything clear in the docs to allow me to do this. Also, it doesn’t shut down.

I’m going to keep hacking around with Axon, it’s pretty neat. Interesting things I’d like to poke around in:

• Replace the underlying IPC mechanism with multiprocessing.Pipe/Pyro/posix_ipc
• hack on the mp version of the component backend
• get a multi-system script running and communicating workloads across the LAN