jessenoller.com

Pre­am­ble

There’s been some recent dis­cus­sion about DjangoCon(.eu | .us) and whether or not speak­ers should have to pay for admis­sion as well — see Chris Wanstrath’s (of Github) tweets (here and here) and this Con­vore thread for exam­ples. Obvi­ously, as PyCon is the “big dog” so to speak for Python con­fer­ences, every­one looks to “us” for a model to work from, or how we man­age things. I’ve seen a lot of poop slung towards the Djan­go­Con orga­niz­ers, mainly due to a lack of know­ing “why” cer­tain poli­cies (such as “Every­one Pays”) exist for Djan­go­Con, PyCon, and other conferences.

As co-chair and pro­gram com­mit­tee chair last year, and pro­gram com­mit­tee chair the year before, and now chair for the next two years — I fig­ured it might be good to take a moment to explain the ratio­nale behind PyCon’s approach — as well as some sta­tis­tics about the bud­get. I’m not going to state that this pol­icy is per­fect; nor that it won’t be changed; I also will not release the bud­get pub­licly — I don’t think giv­ing every­one a spread­sheet with­out the con­text of the hun­dreds of man hours of work that go into it is use­ful, at all.

Impor­tant Note: PyCon is orga­nized and man­aged by the Python Soft­ware Foun­da­tion — this means that, as part of being a 501c3 char­ity, some of the finan­cials from past PyCon is avail­able as part of pub­licly acces­si­ble finan­cial doc­u­ments of the foun­da­tion. You can find those on the PSF’s site.

The same rea­son­ing may not apply to a con­fer­ence that is orga­nized by a com­mer­cial entity or is done for profit. OSCON is a com­mer­cial con­fer­ence, so hav­ing speak­ers get in free is gen­er­ally expected. DjangoCon.us is in the mid­dle — it is orga­nized for profit by a com­mer­cial entity, but it also con­tributes heav­ily back to the Django Soft­ware Foun­da­tion. DjangoCon.eu is man­aged dif­fer­ently as well.

» Read the rest of this entry «

Preamble There's been some recent discussion about DjangoCon(.eu | .us) and whether or not speakers should have to pay for admission as well - see Chris Wanstrath's (of Github) tweets (here and here) and this Convore thread for examples. Obviously, as PyCon is the "big dog" so to speak for Python conferences, everyone looks to ...

Ah, PyCon 2009 has come and gone. I’m a lit­tle late in doing a wrap up — but that’s pri­mar­ily due to the fact I wanted to spend time with the fam­ily, and not on the com­puter since I got back mid-day Thurs­day of last week. That — and other things in my life are a lit­tle worse-for-wear, so my brain’s been full of “other stuff” rather than bask­ing in the glow of:

The best pycon, ever

Yeah, I came out and said it. For me, it was my 4th(?) PyCon, and not to dis­par­age the past, but this one was the best planned, best exe­cuted and most fun for me. Sure, it had a few neg­a­tives (Chicago being one of them), but by far the pluses out­weighed any minuses.

Here’s a sum­mary of things I can remember.

The summits

I got in mid-day wednes­day and broke into the VM sum­mit. I was imme­di­ately slapped with the unladen-swallow announce­ment, which made me do a lit­tle dance and imme­di­ately start pulling down the code. For the most part, since I was past the intro­duc­tion piece, I sat in back and had inter­est­ing dis­cus­sions with Thomas Wouters, Brett Can­non and others.

Later that night, I sat down and spent some time with unladen swal­low — mainly get­ting LLVM all happy and pulling down the trunk (when I should have been using the released tag). I also spent a fair amount of time mas­sag­ing my slides for both of my talks. Noth­ing quite like last minute changes.

The next day was the lan­guage sum­mit. Here a bunch of peo­ple much smarter than me sat down and dis­cussed the future of the lan­guage. I can’t even begin to describe every­thing that hap­pened — some basic high­lights are:

• Some dis­cus­sions around mak­ing python 2.7 be the last release of the 2.x line. I think the gen­eral con­sen­sus was to con­tinue to use 2.x as a ramp into 3.0, but in gen­eral if migra­tion to (and from) 3.0 was faster/easier, then the adop­tion bar­rier for 3.x would be greatly less­ened. Most of us finally agreed that the 2.x line should end with 2.7.
• On 3.0 — port­ing issues and pains were dis­cussed, things like 2to3 being slow, lack of incen­tive (3k being “a bet­ter lan­guage” notwith­stand­ing) being one of the big issues. Some­one (Guido I think) floated the idea that big fea­tures only go into the 3.x branch, ergo pro­vid­ing greater upgrade incen­tive. I don’t think a lot of peo­ple dis­agreed with this and in fact, I took it to heart dur­ing the sprints when look­ing at fea­ture requests for multiprocessing.
• Which brings us to 3to2 — sev­eral devel­op­ers expressed that if we had a tool which could trans­late python3 code to python2, then they could start writ­ing their frameworks/tools in pure python3 and then back port it. I don’t recall any­one dis­lik­ing this idea — but no one wanted to get the ball rolling. So I raised my hand. In essence, I vol­un­teered to get the ball rolling, which I have done. Ben­jamin Peter­son will be head­ing it up pri­mar­ily — but any­one is free to con­tribute (and some already have). It’s got poten­tial as a google sum­mer of code project.
• After that we dis­cussed what CPython can do to make integration/testing eas­ier for alter­na­tive imple­men­ta­tions. We agreed that being able to mark tests as “cpython only” and break­ing things in the repos­i­tory up in such a way to make inte­gra­tion of the stan­dard library into alter­na­tive dis­tri­b­u­tions eas­ier. Brett has more on this, but the essen­tial idea is to com­part­men­tal­ize the stan­dard library out into a sep­a­rate project within source con­trol, and have other imple­men­ta­tion pull/ingrate that. Any­thing that’s Jython/CPython/etc spe­cific would be marked/stored appro­pri­ately. For exam­ple, mul­ti­pro­cess­ing as it exists today is a CPython arti­fact, and not needed for say, Jython.
• And then there was the pack­ag­ing dis­cus­sion. Rather then rehash every­thing that was dis­cussed, I’ll sim­ply men­tion that I pro­posed a much sim­pler approach. Specif­i­cally, I noted that we should sim­plify the core of dis­tu­tils, pulling in setup­tools fea­ture where appro­pri­ate, offer plugin/hooks where appro­pri­ate, stan­dard­ize on a sim­ple, exten­si­ble meta­data for­mat and remove any of the “non core” bells and whis­tles from dis­tu­tils. In essence my argu­ment was that things like RPM build­ing, easy_install, vir­tualenv, and so on sim­ply do not belong in core python, rather they belong out­side of the core, where consumers/developers can tai­lor solu­tions to suit their needs. Ulti­mately disu­tils can help them, and make the APIs/Plugins/Metadata stan­dard but it should not be a fully fledged “pack­age man­ager” or build utility.

Ulti­mately, the sum­mits were a ton of fun, and being lucky enough to be invited was really great for me. At the end I was wor­ried about peo­ple being tired of my chirp­ing up, but it was resound­ingly a success.

The conference

This is where things begin to blur — hop­ping in between hotels, light­ning talks, hall­way dis­cus­sions — it was insane. Add to that my fear/stress about both of my talks, and I sim­ply can’t fit every­thing that hap­pened in my brain.

I think I started the day in “How to Give a Python Talk” by AMK — which was great, if dis­heart­en­ing sim­ply due to the fact that I had made some of the mis­takes he pointed out in my talks. Ergo more slide hack­ing. After that it was hall­way dis­cus­sions and hit­ting up “How Python is devel­oped” by Brett — I was going to heckle him, but I played nice.

Then there was the Python VMs panel, which I swapped out of and into the “Build­ing an Auto­mated QA Infra­struc­ture using Open-Source Python Tools” — which can be summed up as “Yay build­bot” (I’m hard to impress, I’ve built a few of these by now). I then hit up the Twisted/AMPQ talk but ducked out early to go prep for my talk.

Then, it was my mul­ti­pro­cess­ing talk time. You can see the full video here, I think it went well, and feed­back has been pos­i­tive. Given I’ve done vari­a­tions on this talk else­where, I was a lit­tle more con­fi­dent in myself. I think I could have slowed down in some parts, and stopped fid­get­ing as well as a few other nits. I had some great con­ver­sa­tions with users of the pack­age as well as peo­ple new to it which pretty much absorbed all my time until din­ner. I think I hit up the light­ning talks — but I can’t remember.

The next day, was the now infa­mous LINDBERG’D light­ning talk (video here) — for those who don’t know — Yes, Van knew — he even wrote the legal dis­claimer I have in my slides. Basi­cally, Brett and I had stayed up until what — 1am one night laugh­ing about this idea of send­ing Van (who isn’t very threat­en­ing) after com­menters on the inter­net. The day after we stayed up laugh­ing at this (much to the dis­may of our neigh­bors) I hit Van up for some text and let him in on it. No, it wasn’t seri­ous (peo­ple asked). Yes, I was also voted in with a pile of other peo­ple into the PSF. Good times.

After­ward, the guidonote and then know I saw the state of django talk, as well as Jack’s “Class Dec­o­ra­tors: Rad­i­cally Sim­ple” — which was fan­tas­tic (although I saw a vari­a­tion at the boston python meetup the week before). I dropped in on the ORM panel, and the GAE talk (which was dis­ap­point­ing). Then I hit Bob’s “Drop ACID and think about data” which was awe­some. I again ducked out to dou­ble check my stuff for my dis­trib­uted sys­tems talk. Also on sat­ur­day was the “Writ­ing About Python” BoF, which was cool, but in which I may have over-expressed sev­eral cough strong opinions.

The video of my sec­ond talk is here. This one, my lack of con­fi­dence is appar­ent, and I’m still fid­get­ing. I think part of my prob­lem is I like to express with my hands, almost like a spas­tic air traf­fic con­trol man and hold­ing the mic/trying not to fid­get made my body just sort of spaz. This time though, I rushed from my talk into Alex Martelli’s excel­lent “Abstrac­tions as Lever­age” talk. After that, more peo­ple found me and picked my brain.

I can’t stress how awe­some Alex’s talk was — much of my hall­way dis­cus­sion time was spent dis­cussing abstrac­tions (espe­cially around dis­trib­uted sys­tems) with him.

We’ll see how my call for some cohe­sion and a real “django for dis­trib­uted sys­tems” call in my sec­ond talk goes. Every­one I have spo­ken to likes the idea and had a lot of great feed­back, but lord knows I don’t have the time to lead it up right now, maybe soon.

Late sat­ur­day, was the teach me web test­ing BoF which I ducked into, and then past that the Test­ing In Python BoF which pro­vided me much in the way of fun, ideas, dis­cus­sions and uh. Beer. Yeah. I can’t exactly remem­ber what I said when Titus put me on the spot to dis­cuss “what sucks about test­ing in python”. But I do remem­ber hav­ing some great laughs. I think the heck­ling got recur­sive at one point, not sure though.

Sun­day came, and with it, my sorry attempt to sit in the “Func­tional Test­ing Tools in Python” panel — which was thwarted by a 45 minute bloody nose. Appar­ently Titus talked a lot, so I don’t think I missed any­thing. End of main con.

Before I move into the sprints — I want to point out that I had fan­tas­tic hall­way dis­cus­sions with tons of peo­ple — peo­ple I look up to, and peo­ple I’ve never met before. I had excel­lent lunches with groups of peo­ple rang­ing from the highly expe­ri­enced to some­one who has learned python a week before. I can’t say enough about the sim­ple fact that you get to relax, hang out and just talk with all of these peo­ple. Python is defined by it’s com­mu­nity — and ours is pretty awesome.

The sprints

Mon­day morn­ing, I was off to the python core sprint. I imme­di­ately started hack­ing on mul­ti­pro­cess­ing bugs — I think all told, I man­aged to close around 12 or so bugs by the time thurs­day morn­ing came around. The sprint was awe­some sim­ply due to the fact I could look across the room/table and ask any num­ber of core devel­op­ers ques­tions. Mar­tin Van Loewis came to my rescue/aid a few times, and being able to bounce ideas off of every­one is pure unbri­dled awesome.

Hav­ing ded­i­cated face to face time to beat on these bugs and share information/debate things is immea­sur­able. I also got access to snakebite thanks to Trent, who is still work­ing out the kinks. Alas, even hav­ing access did not make cross-platform sup­port (mainly the BSDs) any eas­ier for mul­ti­pro­cess­ing. I spent a good chunk of my time futz­ing with vir­tual machines so I could do fur­ther work — I ended up cut­ting that short because I wanted to fix things at the sprints: not fart around with tool chains.

Other than hack­ing until late at night through the sprints, there’s was quite a fun night where bour­bon flowed, and many par­took — much fun was had by all. There’s noth­ing quite like hav­ing drinks and mak­ing awful threading/process/compiler jokes.

Bug fixes/patch work for multiprocessing:

• multiprocessing/pipe_connection.c com­piler warn­ing (conn_poll)
• test_multiprocessing fails on sys­tems with HAVE_SEM_OPEN=0
• Mul­ti­pro­cess­ing pack­age build prob­lem on Solaris 10
• with lock fails on mul­ti­pro­cess­ing
• Bus error when call­ing .poll() on a closed Con­nec­tion from multiprocessing.Pipe()
• mul­ti­pro­cess­ing: Sock­etLis­tener should use SO_REUSEADDR
• mul­ti­pro­cess­ing queues.py doesn’t include Join­able­Queue in its __all__ list
• patches for mul­ti­pro­cess­ing mod­ule on NetBSD
• test_multiprocessing: test_listener_client flak­i­ness
• mul­ti­pro­cess­ing: switch to auto­conf detec­tion of plat­form val­ues: This one was a bear. All the props in the world go out to Mar­tin Van Loewis for guid­ing me through get­ting this done.
• multiprocessing.Pipe ter­mi­nates with ERROR_NO_SYSTEM_RESOURCES if large data is sent (win2000)
• imple­ment ini­tial­izer for multiprocessing.BaseManager.start()

In Sum­mary

PyCon is sim­ply get­ting bet­ter — I’m look­ing for­ward to Atlanta next year, if noth­ing more than for bet­ter weather. I got to talk to a lot of great peo­ple — Collin Win­ter, Thomas Wouters, Brett Can­non (be care­ful, he’s an odd one), Guido, Alex Martelli and many oth­ers. Hav­ing many hall­way dis­cus­sions with lan­guage users and encour­ag­ing them to get involved — PyCon is Awe­some­Con — no where else do you get to hear so much great infor­ma­tion, dis­cus­sions and points of view.

All of the videos for this pycon are going up on pycon.blip.tv.

It was a great time, and I can’t thank every­one involved enough.

Ah, PyCon 2009 has come and gone. I'm a little late in doing a wrap up - but that's primarily due to the fact I wanted to spend time with the family, and not on the computer since I got back mid-day Thursday of last week. That - and other things in my life ...

Slides from my intro to Concurrency/Distributed sys­tems talk @pycon 2009 are here. Also up on the pycon site here

Slides from my intro to Concurrency/Distributed systems talk @pycon 2009 are here. Also up on the pycon site here

Slides from my intro to mul­ti­pro­cess­ing talk @pycon 2009 are here. Also up on the pycon site here

Slides from my intro to multiprocessing talk @pycon 2009 are here. Also up on the pycon site here

Note:This is another post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more./Note

Stack­less python — here’s another big one on the pile — is much more than a library, or a frame­work which runs on CPython — Stack­less is actu­ally a mod­i­fied ver­sion of the CPython inter­preter. It’s much more than just a C-extension. Stack­less is in use by var­i­ous peo­ple and com­pa­nies — most notably, it’s in use by CCP Games, mak­ers of Eve Online (see this pycon pre­sen­ta­tion). In fact, CCP Games is a large part of why Stack­less is still around today.

» Read the rest of this entry «

Note:This is another post in what I hope will be a series leading up to my concurrency/distributed systems talk at PyCon. I'm steadily working through experimenting with and learning the various frameworks/libraries in the python ecosystem. I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the author(s) ...

I, along with a whole heck of a lot of other peo­ple will be attend­ing PyCon in march. You should know about this by now, unless you’re liv­ing under a rock, or in a shoe­box (I like shoeboxes).

PyCon 09 is turn­ing out to be one of the ones I am most excited about in some time — bar­ring the fact they let me actu­ally stand up and speak about some­thing, there’s a ton of other excel­lent and excit­ing things going on.

I will be doing two talks — “Intro­duc­tion to Mul­ti­pro­cess­ing in Python” on Fri­day, at 3:20 PM, and “Con­cur­rency and Dis­trib­uted Com­put­ing with Python Today” at 3:20 PM Saturday.

The for­mer talk is easy, given it will be focused on intro­duc­ing the mul­ti­pro­cess­ing mod­ule to the masses, and tak­ing ques­tions about it (be gen­tle, I just work here). The lat­ter is a much big­ger beast. I’ve been blog­ging about my research in my pycon 2009 cat­e­gory, and I still have a pile of things to keep adding to that. The talk will attempt to dif­fer­en­ti­ate con­cur­rency from dis­trib­uted sys­tems, and show the var­i­ous toolkits/frameworks/etc in the ecosys­tem today, to help you build both types of systems.

Given both talks are 45 min­utes in length, I will be pub­lish­ing my talk notes (my slides are not going to be heavy weight) and other infor­ma­tion here.

In addi­tion to me speak­ing, which may or may not be excit­ing, there’s one hel­luva ton of other talks which sim­ply look awesome.

My sched­ule looks like this:

• Thurs­day: Python Lan­guage Sum­mit, where I will endeavor to be smart.
• Fri­day: How to give a python talk
• Fri­day: Using Windmill
• Fri­day: Intro­duc­tion to Python Pro­fil­ing or How Python is Devel­oped, to harass Brett.
• Fri­day: Panel — Python VMs
• Fri­day: Build­ing an Auto­mated QA infra­struc­ture using Open Source Tools
• Fri­day: Twisted, AMQP and Thrift: Bridg­ing mes­sag­ing and RPC for build­ing scal­able dis­trib­uted applications
• Fri­day: My Talk (I fig­ure I should go to it)
• Fri­day: A Whirl­wind Excur­sion through Writ­ing a C Exten­sion or Chal­lenges and Oppor­tu­ni­ties for Python
• Thurs­day: Plu­g­ins and mon­key­patch­ing: increas­ing flex­i­bil­ity, deal­ing with inflexibility
• Sat­ur­day: The (lack of) design pat­terns in Python or the Pinax talk
• Sat­ur­day: Class Dec­o­ra­tors: Rad­i­cally Simple
• Sat­ur­day: Panel: Object Rela­tional Map­pers: Philoso­phies and Design Decisions.
• Sat­ur­day: Drop ACID and think about data (I won­der if we can take this lit­er­ally, if he has scary slides though, we all might trip balls)
• Sat­ur­day: My Talk, which is up against Bruce Eckel and Ray­mond H — I expect no one to show up.
• Sun­day: Panel: Func­tional Test­ing Tools in Python
• Sun­day: Design­ing a web frame­work: Django’s design decisions

This doesn’t even cover the open space dis­cus­sions which I might attend — includ­ing the “Writ­ing About Python” one Doug Hell­mann is putting together as well as the “Teach Me Web Test­ing” one by Steve Holden.

After the main con­fer­ence, I’ll be stick­ing around until Thurs­day morn­ing for the sprints, at which point I should be suf­fi­ciently burned out on python stuff, I will fully con­vert over to being a full time burger flipper.

Right now there are 620 reg­is­tered peo­ple who made their atten­dance pub­lic I don’t know how many there are in total.

Hope to see you there!

I, along with a whole heck of a lot of other people will be attending PyCon in march. You should know about this by now, unless you're living under a rock, or in a shoebox (I like shoeboxes). PyCon 09 is turning out to be one of the ones I am most excited ...

Note:This is the third post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more. Addi­tion­ally, thanks to glyph for giv­ing me a hell of a lot of feed­back./Note

Twisted is the 800 lbs gorilla of the “con­cur­rency” frame­works. It’s been around for awhile, has a large fol­low­ing — it’s used by every­one from Apple (iCal server) to Build­bot Build­bot. It has a lit­eral ton of sub projects and other “semi attached appendages”.
» Read the rest of this entry «

Note:This is the third post in what I hope will be a series leading up to my concurrency/distributed systems talk at PyCon. I'm steadily working through experimenting with and learning the various frameworks/libraries in the python ecosystem. I reserve the right (and probably will) to revise these entries based on feedback from people (mainly ...

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on python-list, but more recently, I know he’s been work­ing on get­ting some level of mul­ti­pro­cess­ing into cir­cuits — cir­cuits was already on my research list, but I bumped it up on the queue because we started chatting.

Let me state this: these are slightly cleaned up ver­sions of my notes as I am learn­ing these mod­ules — some of them have higher learn­ing curves for me than others.

Cir­cuits is, well — an event based “frame­work” (again, note the small f) based around the con­cept of Com­po­nents (big C!) consuming/reacting and in turn gen­er­at­ing events — all asynchronously.

James’ goals seem pretty sim­ple — build some­thing with no exter­nal depen­den­cies, that’s com­pact (I should say, the core.py is < 500 lines) and that makes it easy to build scal­able mes­sag­ing based (event based) sys­tems. Did he suc­ceed — don’t know! Let’s dig in.

» Read the rest of this entry «

Next up in the GBLOSTR (great big list of stuff to review) is the Circuits library by James Mills (here and here) I'm familiar only with James Mills' posts on python-list, but more recently, I know he's been working on getting some level of multiprocessing into circuits - circuits was already on my ...

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more. Code and exam­ples will be checked into my pycon 2009 bit­bucket site here/Note

For awhile now, I’ve been mean­ing to dig into Kamaelia but was largely put off by what I tend to call the “twisted effect”. What this means is that when I go look­ing for libraries and small com­po­nents, I go look­ing for a library — not a “solu­tion”. I also worry about the “once you go in, you must fol­low this par­a­digm” effect. I’m not going to say that these feel­ing con­tinue to be founded, or are com­pletely ratio­nal — after all, I am dig­ging into it, yes? It’s the thought that once you adopt the “one true way of doing things” you’re trapped in that solution/framework “for­ever” — iron­i­cally, I love Django for it’s “con­cep­tual integrity” and full-stack approach. No, I don’t under­stand me either.

» Read the rest of this entry «

Note:This is the first post in what I hope will be a series leading up to my concurrency/distributed systems talk at PyCon. I'm steadily working through experimenting with and learning the various frameworks/libraries in the python ecosystem. I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the ...