jessenoller.com

Doug Hell­mann and Mike Driscoll put up an excel­lent post on the Python Soft­ware Foun­da­tion blog about most of the grant-type work that the foun­da­tion per­formed over the 2011 year. To add some color to it — reviews and dis­cus­sions about grants and award­ing this com­prises quite a bit of the board-level work that goes on (exclud­ing indi­vid­ual committees).

You can see from the post quite a bit of the cap­i­tal spent goes to sup­port other con­fer­ences — as I’ve stated before, money that comes into the foun­da­tion in the forms of dona­tions and PyCon “rev­enue” goes back into the sys­tem to be issued out to things like this.

This is why I am so hot to encour­age grants around Port­ing to Python 3 — I think that the PSF can, in the next year, increase grant work for con­fer­ence and out­reach as well as devel­oper work (such as port­ing libraries and other projects). None of these things should be solely focused on CPython alone — PyPy, Jython, etc should all be recip­i­ents of grants.

And therein lies the rub.

The PSF does not “go look­ing” for places to issue grants — the PyPy grant at PyCon 2011 was a bit of an aber­ra­tion in that I pro­posed it to the board directly.

We need appli­ca­tions from the com­mu­nity! We can do things such as cover meetup fees for user groups, or help fund con­fer­ences, or devel­op­ment work. Jes­sica McKel­lar, I and oth­ers recently revamped the PSF grants page to hope­fully pro­vide a bet­ter out­line of how grants work.

If you have more ques­tions — feel free to ask me here or via email — the PSF’s mis­sion is hap­pily broad, and we’re here to serve and rep­re­sent the com­mu­nity as best we can. But we do need to hear from you!

Doug Hell­mann and Mike Driscoll put up an excel­lent post on the Python Soft­ware Foun­da­tion blog about most of the grant-type work that the foun­da­tion per­formed over the 2011 year....

As I said in my post this morn­ing — “2011 in Review: The Per­sonal Por­tion” — it’s that time where we’re all tak­ing stock and reflect­ing back on 2011.

In this post’s case, I’m tak­ing stock of the things that changed for me — things that stick out in my mind and projects I’ve either started, floun­dered or run com­pletely into ground.

Design and Expe­ri­ence Matter

Per­haps the biggest shift for me in Python-as-a-whole is a move­ment more towards the social / man­age­ment aspects. I’m a Python Soft­ware Foun­da­tion board mem­ber, so obvi­ously me need­ing to take a “big­ger view” isn’t that sur­pris­ing. What has been sur­pris­ing to me is that every­where I turn, I see things we as a whole can do better.

Now, before you think I’m about to go off the deep end; let me assure you — I wouldn’t trade the com­mu­nity I’m lucky to be part of for any­thing, as I’ve said more elo­quently before. How­ever, only a fool believes that any­thing is per­fect, and only the insane only focus on the flaws.

Tak­ing a step back, I’ve seen more and more things that I think we can do a bet­ter job at, and these real­iza­tions all revolve around my con­tin­ued “tran­si­tion” from more back-end to more front-end design and cod­ing. As I’ve become more focused on the users/community and those who are new, I’ve grown to inter­nal­ize the fact that design and expe­ri­ence mat­ter not only in code, and in a GUI, but they mat­ter to a com­mu­nity and lan­guage as a whole.

I’ve spent the bet­ter part of this past year focused on issues around this — encour­ag­ing peo­ple to get involved in the “softer” side of things — help­ing out with doc­u­men­ta­tion, men­tor­ship and edu­ca­tion, try­ing to get peo­ple to think more about one another and those just get­ting started and intro­duced to things.

I think that we as a com­mu­nity — and I mean every­one — from Django to Plone, from Twisted to Tor­nado, from PyPy to cPython can take a look at the “more human” aspects and find things to improve. Some­times it requires fresh eyes to show you what’s bro­ken — peo­ple who do code reviews reg­u­larly know this.

For an exam­ple, look at Ken­neth Reitz’ Requests mod­ule — billed as “HTTP for Humans” — this might be a per­fect exam­ple of the point I’m try­ing to get across. Built on top of “less friendly” libraries, it’s API is a joy to use. It’s sim­ple, it’s clear — the doc­u­men­ta­tion is well done and the entire project feels very wel­com­ing. Per­haps “Wel­com­ing” is the best word for what I’m look­ing for.

I get stuck in want­ing to fix “all the things” — and I can’t help but get mired down in the details of how we make every­thing more wel­com­ing and the expe­ri­ence bet­ter, how do we lower the bar­rier and reduce fric­tion. The result is that I’ve bro­ken my promises to myself and taken on more things than I can pos­si­bly hope to do justice.

How do we make things more wel­com­ing, how do we help the new peo­ple, how do we help those of us grow­ing stuck in our ways to find and explore new things? How can we do this as a com­mu­nity to lift us all up? What I think we need is a series of small, pos­i­tive changes. Lit­tle things like, say:

• User friendly READMEs and Doc­u­men­ta­tion. Yes — I said friendly — don’t assume your users are mag­i­cal super smart engi­neers and users. While the arti­cle is more web focused, I enjoyed “The Myth of the Sophis­ti­cated User” — please don’t assume peo­ple are run­ning bleed­ing edge ver­sion of every­thing, and please don’t assume every­one knows 20 years of Python pack­age development.
• Men­tor­ship! Set up some­thing within your project or team that is focused on men­tor­ing peo­ple to a point where that per­son is com­fort­able to be a con­trib­u­tor.
• Stop the vit­riol. If you find your­self angry when you’re typ­ing that reply to a mail­ing list; walk away. If you see oth­ers being hos­tile or just flat out rude, call them out on it (pri­vately first, no rea­son to be a jerk). Aim to be polite and welcoming.
• The next time you’re putting some­thing up on the web? Take a moment to think about or learn about mak­ing some­thing — yes — pretty and usable. Even if it’s some­thing sim­ple, take a moment to real­ize that you’re build­ing some­thing that may be your future user’s first expe­ri­ence with you. It may be as sim­ple as pick­ing up “Design for Hack­ers” (which I quite liked) or just going with some­thing with sane defaults — like twit­ter boot­strap.
• Speak­ing of sane defaults — please be opin­ion­ated. When a new user wants to install some­thing, don’t give them the com­plete his­tory of pack­ag­ing, just gen­tly explain to them how to do it. Even if I don’t agree with the way you do that, it’s a far cry from 20 years of devel­op­ment his­tory being dumped on some­one when a sim­ple pip install <blah> could work. The same goes for your soft­ware: Pick sane, ratio­nal defaults and abstract away as much as you can. Put exam­ples of usage before the API in documentation.
• APIs and syn­tax mat­ter: your com­mu­ni­ca­tions chan­nels to your users are APIs and syn­tax just as much as your actual code and libraries.

Mov­ing on — I hate to say it this way; but think of the Users and tar­get audi­ence. Remem­ber, you — the per­son read­ing this — and I — are in a tiny minor­ity of the pop­u­la­tion where soft­ware (for the most part) isn’t magic, we under­stand his­tory and we’re very tol­er­ant of unfriendly things and fail­ures because that’s how we “grew up”.

Not every­one knows how to build an inter­preter; or a web frame­work — it doesn’t mean they still can’t contribute.

The Python Soft­ware Foundation

As most of you know — I am one of the direc­tors of the Python Soft­ware Foun­da­tion, and have been the past two years. 2011 was another year where the PSF got to do some pretty cool things. I’ve been stress­ing and push­ing more and more that the PSF has to be focused not just on the “IP” of Python, or just on cPython devel­op­ment — we have to take a larger view of the entire com­mu­nity — this means encour­ag­ing projects such as PyPy, out­reach work­shops, con­fer­ences, etc via grants and support.

You should really take a look at the Python Soft­ware Foundation’s blog — Doug Hell­mann, Brian Curtin and oth­ers have done their best to doc­u­ment and show­case what the PSF has been up to, and where we’re try­ing to help.

My pri­mary focus has been encour­ag­ing things such as the Out­reach and Edu­ca­tion com­mit­tee, and work­ing behind the scenes with a lot of peo­ple to improve the Python.org infra­struc­ture. More recently I’ve been work­ing on a project which should hope­fully become pub­lic soon — but is tied to my first point about Design and Expe­ri­ence and the PSF.

I want the PSF to grow in the good works it per­forms — more grants as we can afford it, get­ting bet­ter host­ing for things as needed, help­ing out projects like Read The Docs or help­ing push for­ward Python 3. The PSF is the Python Soft­ware Foun­da­tion — we need and should be sup­port­ing and help­ing every­thing from PyPy to PyPI, cPython to Scipy.

I think the best way for me to help here is to pick up where I left off doc­u­ment­ing the PSF. Once again — the design and inter­face matter.

The Sprints Committee

As part of my board work back in 2010 I helped start the Python Sprints project — and under Brian Curtin’s guid­ance in 2011, it has con­tin­ued to make small dona­tions in places it mat­ters. In 2012, I’d like to see if I can spin back around and help it grow more and flour­ish, per­haps even be able to pro­vide more money where it’s needed. It’s growth has been slow — but that’s also due to us see­ing less sprints over­all it seems.

GetPython3.com

Started as a side project (yes. another one. sigh.) Get Python 3 is meant to serve as a pile of infor­ma­tion and resources about Python 3 — and as many of the aspects of Python 3 as pos­si­ble. Where to get fund­ing, how to port, what is ported. I’ve actu­ally got­ten some excel­lent help from oth­ers (see github) and I’m hop­ing to grow it more. I’ve got­ten pretty good feed­back on it — and I never turn down a patch!

Python (Core) Mentorship

Dri­ven from my expe­ri­ence with the first point about being wel­com­ing, I’ve done my best to spin up the Python Core Men­tor­ship group, a team / list focused on men­tor­ing new peo­ple into con­tribut­ing to core Python. To quote the home page:

The mis­sion of the Python Core Men­tor Pro­gram is to pro­vide an open and wel­com­ing place to con­nect stu­dents, pro­gram­mers – and any­one inter­ested in con­tribut­ing to the Python Core devel­op­ment. This project is based on the idea that the best way to wel­come new peo­ple into any project is a venue which con­nects them to a vari­ety of men­tors who can assist in guid­ing them through the con­tri­bu­tion process, includ­ing dis­cus­sions on lists such as python-dev, and python-ideas, the bug tracker, mer­cu­r­ial ques­tions, code reviews, etc.

While traf­fic is low, I think it has done it’s job — as with every­thing else on my list, I’d like to see growth — as it is, due to every­thing else on my plate, oth­ers have stepped up to help lead and guide the group. As it is, I’ve run into a case where as I’ve found with many other projects like this — peo­ple are already “tapped out” — myself included. More on resource con­tention later — and I should really do a poll and gauge the list for the rel­a­tive level of suc­cess they feel the group has engendered.

Python Speed Project

Another side-burner project is the Speed.python.org project — this one makes me sad(der) than my other time-starved projects. While we have finally been able to set it up as a PyPy build slave and have it feed­ing results to speed.pypy.org (see the speed-python results), it has not taken off as much as I hoped. We have a beast of a machine (see my ini­tial announce­ment) — but we’ve hit the resource wall like every­thing else. Not enough peo­ple with enough time and the right skills.

The Ele­phant in the room: PyCon 2012

My sin­gle biggest project this year has been get­ting PyCon 2012 ready to fly — every­thing from get­ting the new web­site launched, the staff assem­bled, writ­ing a code of con­duct, and pro­vid­ing white-glove ser­vice and sup­port (and get­ting) our amaz­ing list of spon­sors.

I can’t really esti­mate how many hours I’ve “worked” on Python — but I can tell you every hour has been worth it. Even though it’s sucked my time from other things and projects, it looks like it’s going to be an amaz­ing con­fer­ence. We have robots, we have amaz­ing talks, amaz­ing keynote and ple­nary speak­ers (Paul Gra­ham and Stormy Peters for starters). We have awe­some tuto­ri­als and even more to come.

PyCon rep­re­sents the sin­gle biggest “com­mu­nity act” that the Python Soft­ware Foun­da­tion per­forms — not only does the PSF fund PyCon, but it man­ages it, assumes the risk, etc. I wrote about it in detail in my post “Mak­ing the Case for Spon­sor­ship” and in the “Every­body Pays” post. I’m hop­ing to con­tinue to write up more and more of the details of the inner work­ings of PyCon, as I think it’s an impor­tant series of data points and lessons. Remem­ber — any funds “left” from PyCon go the PSF which allow the foun­da­tion to issue grants to other con­fer­ences, to devel­op­ers, groups and work­shops. It helps us help you.

PyCon 2012 is the thing I am most proud of; we have 80 spon­sors and part­ners (Such as Open­Hatch and PyLadies), we have a solid team of orga­niz­ers work­ing together to bring PyCon 2012 to fruition. We have a robust finan­cial aid pro­gram as is tra­di­tion. I can only hope that I have the tenac­ity and will to see it come together and be able to look at a sea of 1500 Python­istas — new and old in Santa Clara.

ps: You can reg­is­ter here. :)

Blood from a Stone

How do you get more time from peo­ple who are busy? Time and Time again, I’ve found myself ask­ing that ques­tion. Each one of the projects I’ve listed has hit the same issue over and over again. How do you get the vol­un­teers nec­es­sary to help? Heck, even my call for help with mul­ti­pro­cess­ing in August fell on a mostly flat note — prob­a­bly due to me.

I no longer feel “ok” ask­ing for help with new projects sim­ply due to the fact that I know every­one is busy — it’s insane of me to ask peo­ple to take their time away from their projects or fam­i­lies or jobs.

What that means how­ever is that I have com­pletely failed in the not-taking-on-new-things depart­ment — and I don’t see this chang­ing much with­out me flat out learn­ing to tell myself “no”. I believe in this com­mu­nity — I believe in the peo­ple, the friends I have, the lan­guage and every­thing involved. It’s not just another tool for me; it never has been. I’m still learn­ing, and mostly fail­ing (or flail­ing, depends on where I’m standing).

Fin­ish­ing this one off

Look­ing at the list I’ve typed out above, I sud­denly have the feel­ing that I didn’t actu­ally do much last year, I know thats wrong (a nasty look from my fam­ily mem­bers would eas­ily remind me of that). I have been able to help out where I can mak­ing things more friendly, more wel­com­ing and to reach out when and where I can to offer help, and support.

I’ve watched the com­mu­nity change in some dra­matic ways, I’ve looked on as PyPy has gained amaz­ing momen­tum, more and more ven­dors and com­pa­nies have come out with Python sup­port and stat­ing that they’re using Python (and are hir­ing). I’ve got­ten to work with PSF mem­bers, the board, and many, many oth­ers — all I can do is keep at it, and hope I do things justice.

As I said in my post this morn­ing — “2011 in Review: The Per­sonal Por­tion” — it’s that time where we’re all tak­ing stock and reflect­ing back on 2011. In this...

I just banged my head against this, and with no good answers float­ing around out there, I thought I’d share. In my case, I just wanted to extend the basic django.contrib.auth.forms.UserCreationForm in order to make it so when a user was added, an email address had to be sup­plied in addi­tion to the user­name and pass­word fields.

Here is a work­ing exam­ple (forms.py) — just so I don’t for­get it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from django import forms
from django.contrib.auth.models import User
from django.contrib.auth.forms import UserCreationForm
 
class UserCreateForm(UserCreationForm):
    email = forms.EmailField(required=True)
 
    class Meta:
        model = User
        fields = ("username", "email", "password1", "password2")
 
    def save(self, commit=True):
        user = super(UserCreateForm, self).save(commit=False)
        user.email = self.cleaned_data["email"]
        if commit:
            user.save()
        return user

You have to mod­ify the save method on the form to add the email to user object returned by the super call. You can use this to expose other fields on the User object as needed.

I just banged my head against this, and with no good answers float­ing around out there, I thought I’d share. In my case, I just wanted to extend the basic...

Recent posts and dis­cus­sions around port­ing of exist­ing libraries and frame­works to Python 3 have been pretty inter­est­ing. I think that there have been a lot of good points brought up in the dis­cus­sion (See: Armin’s Post (and fol­lowup), Nick’s entry on Python 3 and Nick’s email to Python-Ideas).

On a per­sonal level; I’ve felt frus­trated that there’s not much that I can do myself — I do believe that 2.7 is the proper end of the road of Python 2, and I do think that Python 3 is the future of the lan­guage. Does that mean Python 3 is per­fect? Oh hell no. Does it mean that we can do work to make Python 3 the “Python 3″ we all want and need?

Yes it does.

So; while there is noth­ing I can do directly other than con­tinue to work on the site I’ve been slowly build­ing — GetPython3.com with help from the com­mu­nity — there is an aspect I can help with from a Python Soft­ware Foun­da­tion / Grants level. That means money (well, not unlimited).

As some of you might know — the PSF has actu­ally issued grants to devel­op­ers who have applied to port impor­tant libraries to Python 3 — as I say on the GetPython3 page:

In short: yes — there’s a bevy of infor­ma­tion, videos and blog posts out there that can help you on your way. Python 3 is the future of the Python lan­guage, and enti­ties such as the Python Soft­ware Foun­da­tion strongly believe in sup­port­ing the port­ing effort.

For exam­ple, the Python Soft­ware Foun­da­tion has issued devel­oper grants to port projects such as the email pack­age, PyOpenSSL, and WebOb. It has also pro­vided devel­oper grants for other gen­eral Python devel­op­ment work, such as to Brett Can­non that allowed him to com­pletely revamp the Python developer’s guide.

The Python Soft­ware Foun­da­tion is here for not just CPython, or python-core, or python-the-language. It is here for Python — the com­mu­nity, it’s efforts, its devel­op­ers, design­ers and people.

Cer­tain projects — most notably PyPy — have already started dona­tion pro­grams to help fund large-scale devel­op­ment efforts to Python 3. Oth­ers may soon follow.

Addi­tion­ally to the grants-to-developers aspect — the PSF Sprints project has been issu­ing grants for Python sprints in gen­eral, which means you can apply / ask for a grant for a port-to-python3 work­shop or sprint any time!

But; back to where I was going…

My offer to you, the com­mu­nity is this — I can not guar­an­tee you will get a grant, or fund­ing — but what I can do, and what is within my power as a fel­low mem­ber and PSF Direc­tor is offer to help write, and review appli­ca­tions to the PSF Board of direc­tors for grant applications.

That’s right — I will assist you in writ­ing an appli­ca­tion that will be sub­mit­ted to the PSF Board for approval, for grants aimed at port­ing libraries or frame­works to Python 3; or doing spe­cific doc­u­men­ta­tion / core work for Python 3. I can help you write it; pro­vide tem­plates, dis­cuss it with you (I may have some elves help me) and ulti­mately help you put it in front of the board for approval.

Obvi­ously; the PSF does not have unlim­ited funds; nor can it spend funds irra­tionally. Python 3 is impor­tant how­ever — crit­i­cally so — and while we can not fund every­thing, we can do what we can. I am aim­ing at libraries/frameworks which are in wide­spread use (e.g. notable) and that other projects/libraries/frameworks depend on heav­ily (for exam­ple, see the Py3k poll).

Before get­ting started, you should read the basic PSF Grant guide­lines and you should look through the infor­ma­tion on http://getpython3.com/.

If you are inter­ested in this; drop an email to jnoller@python.org — I don’t promise imme­di­ate up-to-the-second turn around — I’ve obvi­ously got a lot on my plate right now, but I will do my best to help.

Recent posts and dis­cus­sions around port­ing of exist­ing libraries and frame­works to Python 3 have been pretty inter­est­ing. I think that there have been a lot of good points brought...

The dead­line for PyCon 2012 tuto­r­ial, talk, and poster pro­pos­als is under 14 days away, so be sure to get your sub­mis­sions in by Octo­ber 12, 2011 (as always, if it’s Octo­ber 12th any­where in the world, sub­mis­sions are still open!).

Whether you’re a first-timer or an expe­ri­enced vet­eran, PyCon is depends on you, the com­mu­nity, com­ing together to build the best con­fer­ence sched­ule pos­si­ble — PyCon is first and fore­most about the com­mu­nity, dri­ven by vol­un­teers both on an orga­ni­za­tional level, and by speakers.

Our call for pro­pos­als lays out the details it takes to be included in the lineup for the con­fer­ence in Santa Clara, CA on March 7–15, 2012. I should note that this year we have gone away from con­sis­tent groups of invited speak­ers — mean­ing, all talks, regard­less of who sub­mits them are not guar­an­teed a speak­ing slot. All talks are judged and reviewed on the mer­its of the talk and the speak­ers themselves.

If you’re unsure of what to write about, our recent sur­vey yielded a large list of poten­tial talk top­ics, and plenty of ideas for tuto­ri­als. We’ve also come up with gen­eral tips on pro­posal writ­ing to ensure every­one has the most com­plete pro­posal when it comes time for review. As always, the pro­gram com­mit­tee wants to put together an incred­i­ble con­fer­ence, so they’ll be work­ing with sub­mit­ters to fine tune pro­posal details and help you pro­duce the best sub­mis­sions. Even if you are still incu­bat­ing a talk idea: sub­mit the pro­posal now in rough form and we can assist you in flesh­ing out and refin­ing the pro­posal dur­ing the review process.

We’ve had plenty of great news to share since we first announced the call for pro­pos­als. Paul Gra­ham of Y Com­bi­na­tor was recently announced as a keynote speaker, mak­ing his return after a 2003 keynote. David Bea­z­ley, famous for his mind-blowing talks on CPython’s Global Inter­preter Lock, was added to the ple­nary talk series.

Spon­sors can now list their job open­ings on the “Job Fair” sec­tion of the PyCon site as we pre­vi­ously announced — pro­vid­ing an excel­lent resource for job seek­ers, and providers.

We’re hard at work to bring you the best con­fer­ence yet, so stay tuned to PyCon news at the PyCon blog and on Twit­ter at https://twitter.com/#!/pycon.

We recently eclipsed last year’s spon­sor­ship count of 40 and are cur­rently at a record 54 orga­ni­za­tions sup­port­ing PyCon. If you or your orga­ni­za­tion are inter­ested in spon­sor­ing PyCon, we’d love to hear from you, so check out our spon­sor­ship page.

And as always — quick thanks to all of our awe­some PyCon 2012 Sponsors:

• Dia­mond Level: Google and Drop­box.
• Plat­inum Level: New Relic, Sur­vey­Mon­key, Microsoft, Eventbrite, Nasuni and Gondor.io
• Gold Level: Walt Dis­ney Ani­ma­tion Stu­dios, CCP Games, Lin­ode, Enthought, Canon­i­cal, Dot­cloud, Log­gly, Revsys, ZeOmega, Bitly,ActiveS­tate, Jet­Brains, Cak­tus, Dis­qus, Spo­tify, Snoball, Evite, Plaid­Cloud and Mozilla
• Sil­ver Level: Imag­i­nary Land­scape, Wis­er­To­gether, Net-ng, Olark, AG Inter­ac­tive, Bit­bucket, Open Bas­tion, 10Gen, gocept, Lex Machina,fwix, github, toast dri­ven, Aarki, Thread­less, Cox Media, myYear­Book, Accense Tech­nol­ogy, Wing­ware, Fresh­Books, and Big­Door
• Lan­yard: Dreamhost
• Sprints: Red­dit
• FLOSS: OSU/OSL, Open­Hatch

Thank you — and as always, feel free to reach out to the team or any of the staff with any ques­tions you might have.

• The PyCon Orga­niz­ers — http://us.pycon.org/2012
• Jesse Noller — Chair­man — jnoller@python.org
• Brian Curtin — Pub­lic­ity Coor­di­na­tor — brian@python.org

The dead­line for PyCon 2012 tuto­r­ial, talk, and poster pro­pos­als is under 14 days away, so be sure to get your sub­mis­sions in by Octo­ber 12, 2011 (as always, if...

PyCon 2012 already has a record-breaking 52 spon­sors! I can not thank every one of them enough (but I will give my thanks again at the end of this post indi­vid­u­ally), and we are always look­ing for more spon­sors to join the ones we have.

I wanted to take a moment to explain what makes spon­sor­ship good for the com­mu­nity, and a sound invest­ment for spon­sors new and old, prospec­tive and future.

This year, as chair, I’ve taken it upon myself to push and man­age PyCon spon­sor­ship (cor­po­rate, non profit, media, etc) for a vari­ety of rea­sons. First, as some­one who has been a spon­sor in the past (and present) and as some­one who spends a lot of time “sell­ing” the Python Soft­ware Foun­da­tion, and the com­mu­nity to oth­ers — I feel very closely tied to PyCon and sponsorship.

Not to men­tion — cor­po­rate spon­sor­ship is what allows us to keep this prob­a­bly one of the least expen­sive inter­na­tional tech­ni­cal con­fer­ences you could pos­si­bly attend this upcom­ing year. With­out spon­sor­ship — and the array of spon­sors we have right now for PyCon 2012, the con­fer­ence could sim­ply not hap­pen at the size it has reached, or have a robust finan­cial aid pro­gram, keep tick­ets and tuto­ri­als cheap, etc. We have, once again inten­tion­ally capped atten­dance at a level to allow for this, and to help keep PyCon’s com­mu­nity feel and closeness.

Run­ning a con­fer­ence is, frankly, a dan­ger­ous game. As I noted in my blog post sev­eral months ago dis­cussing some of the finan­cial work­ings of PyCon and its finan­cial phi­los­o­phy. It is very easy to lose a lot of money, very quickly. PyCon is held / financed / backed by the Python Soft­ware Foun­da­tion. This means lack of spon­sor­ship, low atten­dance, etc could — with a sim­ple mis­step — bank­rupt the foun­da­tion. Spon­sor­ship helps shore up the gam­ble you make sign­ing con­tracts on cater­ing, room book­ings, rental of the space where the con­fer­ence is held, audio/video costs, etc. Although, if you make a big enough mis­take — noth­ing will pre­vent things from going south. This means care­ful plan­ning, bud­get­ing and negotiation.

Also, while PyCon has always been, and will con­tinue to be a com­mu­nity focused and there­fore, low cost and inclu­sive con­fer­ence, not really focused on prof­it­ing from atten­dees, any rev­enue that comes out of PyCon (profit, if you will) goes directly to the Python Soft­ware Foun­da­tion. This money, in turn, is used to improve infra­struc­ture of Python resources, pro­vide devel­oper grants for pro­gram­ming work, pro­vide grants to con­fer­ences all over the world and many other com­mu­nity projects.

In the last few months alone the PSF has issued grants to PyTexas, EuroPy­thon, Python Ire­land, PyCon India, and many, many oth­ers. We have issued grants for port­ing mod­ules to Python 3, ser­vice such as Read The Docs, etc. Any revenue/profit is flipped back into fund­ing PyCon, and the com­mu­nity as a whole.

PyCon pro­vides a very tan­gi­ble entity for cor­po­rate spon­sors — it’s an eas­ier “sell” than direct PSF spon­sor­ship, and there­fore is a fun­da­men­tally bet­ter con­duit for funds into the PSF.

That’s all fine you say: those are great things for the com­mu­nity, and con­fer­ence — but why would a com­pany want to spon­sor PyCon? Sponsors receive tan­gi­ble ben­e­fits such as recruit­ing at the con­fer­ence, adver­tis­ing and mar­ket­ing, get­ting com­mu­nity involve­ment known (call it com­mu­nity karma), etc. Spon­sor­ship isn’t just a mat­ter of ask­ing a com­pany to fund the con­fer­ence because “it’s good for the com­mu­nity” — it’s a mat­ter of show­ing them that not only is it good for the com­mu­nity — it’s good for their goals and needs.

PyCon is an excel­lent recruit­ment tool.

If you’re look­ing for Python pro­gram­mers, a venue filled with 1500 Python hack­ers of all types — from web devel­op­ers, to design­ers, to dis­trib­uted sys­tems engi­neers and oper­a­tions peo­ple is an excel­lent place for you and your com­pany to find “that spe­cial some­one”. I know a lot of Python hack­ers out there who have been hired by com­pa­nies they “met” at PyCon. I also know a lot of speak­ers and tuto­r­ial teach­ers who have received jobs or job offers after speaking/teaching at PyCon.

Just as PyCon is an excel­lent venue for com­pa­nies look­ing to hire, robust spon­sor­ship allows peo­ple at the con­fer­ence know what com­pa­nies out there could be hir­ing Python hack­ers. Com­pa­nies like Walt Dis­ney Ani­ma­tion Stu­dios, Google, Drop­box, and oth­ers as well as com­pa­nies that aren’t well known for being Python shops. It’s a great venue for job seek­ers to find employers.

The Jobs Fair page we added this year for spon­sors, and those look­ing for jobs is a log­i­cal exten­sion of this. Any­thing we can do to con­nect peo­ple and com­pa­nies is great.

PyCon is an excel­lent mar­ket­ing tool.

If you are look­ing to sell some­thing — an edi­tor, host­ing, a ser­vice, etc — PyCon’s 1500 attendee pool pro­vides an amaz­ing cross sec­tion of peo­ple. Not just hard core devel­op­ers — entre­pre­neurs and startup founders, IT busi­ness peo­ple and lead­ers. Python is a lan­guage that as time goes by — I am less and less sur­prised where it pops up — and more sur­prised when it isn’t being used some­where within a company.

It is lit­er­ally every­where — a fre­quently unsung hero for many com­pa­nies. Some­times, com­pa­nies use it with­out even know­ing it.

Python — and it’s com­mu­nity — and there­fore PyCon is amaz­ingly diverse. This means when you spon­sor PyCon, you are adver­tis­ing to an amaz­ingly diverse group of peo­ple. Skill sets from all walks of tech­nol­ogy — and a sur­pris­ing num­ber of peo­ple to whom Python is a tool they use pro­lif­i­cally to get some other job done (like say, video ren­der­ing or con­trol­ling robots). PyCon’s atten­dees reflect the stun­ning makeup of it’s com­mu­nity. You can’t go wrong get­ting your com­pa­nies names on attendee’s lips.

PyCon is a great way to raise visibility.

This is as much a sub-point of my pre­vi­ous note on mar­ket­ing as any­thing else but it deserves some atten­tion. If you’re a com­pany who is try­ing to get the word out, try­ing to spread the news about your new prod­uct or ser­vice, peo­ple notice PyCon spon­sors. Not only are you listed on the web­site, you get signs, booths and entries in the pro­gram guide at the con­fer­ence. It can be en excel­lent tool for buzz and dis­cus­sion about and launch­ing a new prod­uct or service.

Even if you’re not sell­ing some­thing — and you just want to get the word out about your company’s open source efforts, opin­ion and ideas and use of Python — PyCon is a fan­tas­tic plat­form to do so. It can lit­er­ally be a plat­form you use to launch you name and brand into the community’s shared mind.

PyCon spon­sor­ship breeds good will.

I wish I had stud­ies to show it, but peo­ple within the com­mu­nity and at the con­fer­ence itself see com­pa­nies spon­sor­ing PyCon and under­stand that while those com­pa­nies might be sell­ing, mar­ket­ing or recruit­ing — are still doing the com­mu­nity a huge favor by act­ing as spon­sors. As I said before — the com­mu­nity ben­e­fits are many, just as the spon­sor ben­e­fits are. I can not stress this point enough — the com­pa­nies that help PyCon via spon­sor­ship or atten­dance do it for many rea­sons — some of them finan­cial, but the social aspects are some­thing all of our spon­sor from the past can attest to. Python is an open source lan­guage, with a strong open source ethos run­ning through its com­mu­nity — and see­ing com­pa­nies give back both through code and finan­cially means a lot to every­one in the com­mu­nity — even other sponsors.

PyCon spon­sors help set an exam­ple for the com­mu­nity in terms of involve­ment and support.

PyCon spon­sor­ship is a good, sim­ple and cost-effective investment.

I wish all con­fer­ence had spon­sor­ship pack­ages as cheap and as robust as the ones PyCon has out­lined in it’s prospec­tus. Heck — a good recruiter to find tal­ent can cost a com­pany $30,000 or more alone — by com­par­i­son, the spon­sor­ship lev­els and prices PyCon has are fan­tas­tic deals (espe­cially when you fac­tor in that com­pa­nies under 25 peo­ple can get a 50% off dis­count on two of those lev­els). For less than a price of a good com­puter and mon­i­tor — you can be a Sil­ver spon­sor. For less than the price if you include the desk and fur­ni­ture or soft­ware licenses? A Gold spon­sor. For less than the price of a good recruiter, or Google Ad cam­paign? You can be a Plat­inum or Dia­mond spon­sor and reach out to not just PyCon atten­dees but to the entire Python community.

PyCon is a pro­fes­sional event.

I swell with pride stand­ing in the shoes of the con­fer­ence chairs that have come before me. PyCon, while focused on the com­mu­nity, the lan­guage, learn­ing, teach­ing, being a ton of fun for all of its atten­dees, and excel­lent loca­tion to hack and net­work is one of the most friendly-yet-professional con­fer­ences I have ever had the priv­i­lege to attend.

PyCon is backed by the Python Soft­ware Foun­da­tion — but it is run by vol­un­teers — even I, as chair, am not paid. For all of us involved, it’s a labor of love. It is a way for us to give back to the com­mu­nity, ecosys­tem and com­pa­nies and spon­sors attend­ing or spon­sor­ing. And while it may be vol­un­teer based — it’s 100% pro­fes­sional. From the web­site, to the pro­gram guide, from talk selec­tion and booth assign­ment — every­thing is treated with sin­cer­ity, respect and trust.

Spon­sors can look at PyCon not just as a good invest­ment, or plat­form — but as a safe one — and if they can not, I have failed as chair of the con­fer­ence. The same applies to every sin­gle attendee.

But too much of a good thing?

As with all things, there is a flip side to this. Spon­sor­ship is great for spon­sors, and the com­mu­nity — but PyCon is fun­da­men­tally com­mu­nity focused, and hence we must walk a line between hav­ing robust spon­sor­ship pack­ages, and going the “full spon­sor­ship monty” so to speak. This means that to this day, I hold firm on the pol­icy that spon­sor­ship does not guar­an­tee or pro­vide tuto­r­ial or speak­ing slots to any sponsor.

At PyCon, we are all equals, espe­cially when it comes to talks. Joe devel­oper from nowhere, Antarc­tica can sub­mit a talk, tuto­r­ial or poster ses­sion as can Bob the devel­oper from a Dia­mond spon­sor and they have equal chances of being accepted. If the talk is good, if the speaker is known to be a good speaker, if the con­tent and sub­ject are com­pelling, a pro­posal will be accepted on its mer­its (but even then we can not accept all the deserv­ing ones).

Other con­fer­ences guar­an­tee speak­ing slots for spon­sors — I feel this runs counter to the PyCon ethos and com­mu­nity phi­los­o­phy. Not only are we open in our source, we treat each other as equals and with respect. Ours is the mer­i­toc­racy of ideas and work — and this point can not get lost or for­got­ten in our — my — work on our spon­sors’ behalf to increase the value and return on invest­ment they see.

We also try to keep the adver­tis­ing and vis­i­bil­ity at the con­fer­ence taste­ful — lim­it­ing ban­ner sizes and loca­tions, focus­ing on the ven­dor area expe­ri­ence while also giv­ing spon­sors free admis­sions to the entire con­fer­ence so they too can par­take in the learn­ing, hack­ing and net­work­ing. We find this to be a good bal­ance between the needs of the atten­dees and the needs and desires of the sponsors.

Trust me, if I thought walk­ing around in a NASCAR-like track suit cov­ered in logos would help our spon­sors, I just might — ask the other staff! But that’s just me.

In clos­ing — I want to encour­age you and com­pa­nies you know or work for, to take part in PyCon and get involved. Even if you can not, or do not want to be spon­sors, I encour­age you to sub­mit pro­pos­als, light­ning talks when the con­fer­ence comes, attend the sprints, and recruit on the “down-low” by just talk­ing and hack­ing with everyone.

I encour­age you, and will work with you day and night to join us as spon­sors — but I value your involve­ment in the com­mu­nity, and the con­fer­ence more. Even by just attend­ing, you are enrich­ing us all. If you have sug­ges­tions on how to make spon­sor­ship bet­ter for spon­sors — or gen­eral com­ments or con­cerns, feel free to email me.

Giv­ing thanks

Finally, I’d like to thank all of our cur­rent spon­sors — and an a yet-to-be-named mys­tery sponsor:

• Dia­mond Level: Google and Drop­box.
• Plat­inum Level: New Relic, Sur­vey­Mon­key, Microsoft, Eventbrite, Nasuni and Gondor.io
• Gold Level: Walt Dis­ney Ani­ma­tion Stu­dios, CCP Games, Lin­ode, Enthought, Canon­i­cal, Dot­cloud, Log­gly, Revsys, ZeOmega, Bitly, ActiveS­tate, Jet­Brains, Cak­tus, Dis­qus, Spo­tify, Snoball, Evite, and Plaid­Cloud
• Sil­ver Level: Imag­i­nary Land­scape, Wis­er­To­gether, Net-ng, Olark, AG Inter­ac­tive, Bit­bucket, Open Bas­tion, 10Gen, gocept, Lex Machina, fwix, github, toast dri­ven, Aarki, Thread­less, Cox Media, myYear­Book, Accense Tech­nol­ogy, Wing­ware, Fresh­Books, and Big­Door
• Lan­yard: Dreamhost
• Sprints: Red­dit
• FLOSS: OSU/OSL, Open­Hatch

And of course, if you want more infor­ma­tion on spon­sor­ship — visit the PyCon 2012 Spon­sor­ship page.

PyCon 2012 already has a record-breaking 52 spon­sors! I can not thank every one of them enough (but I will give my thanks again at the end of this post...

Orig­i­nally, this post was going to be much more dif­fer­ent than what is has become — the orig­i­nal title was “Fail­ing in Pub­lic” — but I don’t think “fail­ing” is fair to me per­son­ally, or to any­one who has ever helped me, or con­tributed a patch or a fix to the mul­ti­pro­cess­ing module.

Yes­ter­day, I made a state­ment on twit­ter:

I am offi­cially look­ing for some­one to take over mul­ti­pro­cess­ing main­te­nance from me. http://bugs.python.org/issue6721

Ignor­ing any com­ments in that bug; I main­tain that a later tweet is still true:

Some­times good points and poignant crit­i­cism can be buried in a pile of crap.

In hind­sight; I could have worded the orig­i­nal mes­sage dif­fer­ently “tak­ing over main­te­nance” means that I am, and always have been the sole con­trib­u­tor to the mul­ti­pro­cess­ing code base, which is patently false. Antoine, and many other python core devel­op­ers, and peo­ple within the com­mu­nity have sub­mit­ted bug reports, patches, tests and doc­u­men­ta­tion. My words were inten­tion­ally harsh — but the direc­tion of that harsh­ness was to me; I feel that as the “leader” (for some mea­sure­ment of “lead”) I have been remiss in my respon­si­bil­i­ties and leadership.

Sure; I could be less harsh on myself — but the level of expec­ta­tional debt that I’ve incurred against myself for the mod­ule and the main­te­nance has grown, and grown. Even if I find myself lead­ing PyCon, busy as a PSF Direc­tor, push­ing the core-mentors pro­gram, the sprints pro­gram, and a lot of other com­mu­nity projects, I am still respon­si­ble for the care and feed­ing for the crea­ture I helped cre­ate and birth. I’ve com­mit­ted the sin of “going dark”.

For some his­tory, see:

• Mul­ti­pro­cess­ing in Hindsight
• Python 3.0, some mul­ti­pro­cess­ing info, admin­is­tra­tive notes.
• The 2.6/3.0 Mul­ti­pro­cess­ing todo list
• Pro­gram­mer Inse­cu­rity and Mea Culpa

Months ago — I spun up the multiprocessing-sig mail­ing list, in hopes to engage more peo­ple — highly active users, inter­ested peo­ple, etc to help me pay down the debt. Of course, in ret­ro­spect; it’s unfair for me to expect any­one but me to help me pay down the debt I’ve incurred. On the other hand, the respon­si­ble thing for me to do — the mature thing for me to do — is to ask for help — not to “wash my hands” of any­thing, but rather to take this as an oppor­tu­nity to look as mul­ti­pro­cess­ing as some­thing greater than what I orig­i­nally envi­sioned and sub­mit­ted to core.

I hear from peo­ple every day who are using the mod­ule — every day, some­thing I helped birth helps peo­ple get things done. Mul­ti­pro­cess­ing has grown up by virtue of becom­ing part of Python core, and daily — despite the bugs, the debt and the quirks — it helps devel­op­ers achieve some­thing they might have oth­er­wise been unable (or at least, had a more dif­fi­cult time) to do.

The mod­ule is expan­sive — it has pools, tools for dis­trib­uted pro­gram­ming via man­agers, pipes for inter­process com­mu­ni­ca­tions, it’s fea­ture set is both large, and ulti­mately com­plex in its under­pin­nings. That com­plex­ity — that fea­ture set — is the rea­son why that debt, the bugs, the quirks has grown over time. If it wasn’t being used — I wouldn’t have so many emails about it — or bugs filed against it.

So where are we/it today?

Today, mul­ti­pro­cess­ing has wide­spread usage — in Python 3, there’s actu­ally a new mod­ule named concurrent.futures that builds on the build­ing blocks of mul­ti­pro­cess­ing and thread­ing. Pack­ages like Cel­ery use it exten­sively (and work around inter­nal quirks). For Python 3 — the sky is the future for what mul­ti­pro­cess­ing could be — addi­tional func­tion­al­ity, mov­ing parts of it (such as the pool abstrac­tions) into the con­cur­rent name­space, extend­ing and improv­ing the Man­ager classes, etc. For Python 2.7 — bug fixes, doc fixes only.

If you search the Python bug tracker for the word “mul­ti­pro­cess­ing” regard­less of assignee, you’ll get 119 hits. That’s right; 119 — not all of them are mul­ti­pro­cess­ing bugs — and many of them are dupes, or fixed in recent ver­sions. What that query gets you is an idea of the debt that has to be paid down and resolved. Each one of those bugs needs to be looked at, repro­duced, de-duped and patched. Some of them may be doc­u­men­ta­tion issues, some are pretty hairy (like the afore­men­tioned http://bugs.python.org/issue6721 as well as http://bugs.python.org/issue4106 and http://bugs.python.org/issue8713).

What I ask­ing for — rather than wash­ing my hands of any­thing, or any attempt to absolve myself of respon­si­bil­ity, is for help. I am stretched thin — too thin to do this myself, or to be the only per­son who can main­tain, under­stand or work on this mod­ule. It’s too big for that, it’s too impor­tant for me to be the arbiter of it any longer. It’s big­ger than me.

Aside from the bug queue; there’s a short list of things that need to be done — the docs need to have a fresh, hard set of eyes on them, there are things (behav­iors, fea­tures) that are undoc­u­mented. The test suite needs a com­plete over­haul — when I inher­ited the code, this is the first thing I should have done — but I didn’t. The prob­lem is that the test suite is mired in magic and com­plex­ity, and with­out an expan­sive, main­tain­able test suite, I don’t feel con­fi­dent that the bug list can be addressed with confidence.

So, I come to you with my hat in my hands, a hum­bled man. It’s unrea­son­able for me to ask for oth­ers to “pay down” the debt I’ve incurred; but it’s irre­spon­si­ble, imma­ture and mis­guided of me to think that I alone, or any sin­gle per­son can go at this alone. So I need your help — and, if in time, some­one choses to be the “leader” for the mod­ule, then I will gladly step back. Until then, I will try to con­tinue to be a guid­ing hand and at least point peo­ple in the proper direc­tion, com­mit patches, etc.

If you are inter­ested; please speak up — or just wade into the bug queue. You can sign up to the multiprocessing-sig list, and ask ques­tions there, or if you’re new to core Python, and want some addi­tional men­tor­ship, check out the Python Core Men­tor­ship pro­gram — that list serves as a gen­tle and polite, wel­com­ing intro­duc­tion to core devel­op­ment. No ques­tion — no mat­ter how green — is off lim­its, and it’s already got an excel­lent track record of help­ing peo­ple get up to speed.

In clos­ing; I’m going to apol­o­gize — we all know lives change, careers change, and inter­ests change. All of these things have hap­pened to me, but in chang­ing so quickly and tak­ing on dif­fer­ent roles, I left some­thing impor­tant behind. In doing so, I have done a great dis­ser­vice to you, the com­mu­nity and users.

I will also thank you; with­out you — the users, cur­rent and future helpers, mul­ti­pro­cess­ing wouldn’t exist or be rel­e­vant in any con­text. With­out you, I wouldn’t have the drive to even write this post, fight to get mul­ti­pro­cess­ing into the std lib to begin with, or per­form any of the other roles I do.

So; thank you.

Orig­i­nally, this post was going to be much more dif­fer­ent than what is has become — the orig­i­nal title was “Fail­ing in Pub­lic” — but I don’t think “fail­ing” is...

Yes; of course I upgraded to OSX Lion on day one. To quote myself from twitter:

I am happy in the warm cozi­ness and stark white­ness of Steve Jobs’ monoculture.

Regard­less of that though, I had very few hic­cups with Lion itself — but a few things you need to deal with going in:

• Be on a high band­width con­nec­tion. The down­loads you have to make are huge.
• XCode 4.x is now free in the App Store — you need this — the first thing you need to install after Lion is the lat­est ver­sion of XCode from the App Store, if you do this, all your vir­tualenvs, your home­brew envi­ron­ment (at least mine) just Keep Work­ing. Save your­self some pain.
  • Side note: @kennethreitz, gen­tle­man scholar, has actu­ally done a cus­tom osx-gcc-installer — this con­tains a sys­tem install of GCC and all the good­ness you need (such as install_name_tool) for Python hack­ing. So you might be able to skip the mas­sive XCode install.
  • When installing XCode, for some unknown unholy rea­son, if you have not quit itunes, and itunes helper (see activ­ity mon­i­tor) prior to start­ing the XCode installer, the install will hang. Do your­self a favor and kill it with fire.
  • Remem­ber; the binary direc­tory for the dev tools is in /Developer/usr/bin/ — this includes gcc-4.2
  • Do your­self a favor, drop “export ARCHFLAGS=”-arch x86_64”” into your .bash_profile.
• If you’re run­ning home­brew; after the upgrade, I rec­om­mend a brew update && brew upgrade
• If you use mer­cu­r­ial — you need to install the updated ver­sion found here.
• Just for good mea­sure, do a global (sudo) rein­stall of vir­tualenv, vir­tualen­vwrap­per and pip. Make sure they’re pointed at the right default Python (in my case the sys­tem one).
• If you are using VMWare Fusion: You really need to be run­ning the lat­est ver­sion of 3.x.
• If you are using Boot­camp, and plan on turn­ing on full disk encryp­tion, see this note from Hacker News (this is why I con­fine win­dows to VMware images)

Other than the above; my dev envi­ron­ment pretty much just kept rock­ing — Lion’s default Python install is a healthy Python 2.7.1 — dou­ble nice++ — MacVim, edi­tor of choice just kept plug­ging away, although I have not tried the “offi­cial unre­leased” ver­sion they have for Lion. I have a slight aver­sion to run­ning beta builds of my editor.

Some of you are going to run into some annoy­ances; I can’t help you with all of them, but I can help you with the two inter­face changes I could not deal with (I actu­ally like all the other ones).

• First; go into sys­tem pref­er­ences > mis­sion con­trol and uncheck “Auto­mat­i­cally rearrange spaces based on most recent use” — trust me, you’ll thank me.
• Sec­ond, there’s this … annoy­ing ani­ma­tion when you make new appli­ca­tion win­dows (See the Ars Lion Review | kin­dle ver­sion). The ani­ma­tion offends me on a cel­lu­lar level. You dis­able it by run­ning this on the com­mand line:

  defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool NO ; killall Dock

• Show hid­den files in finder:

  defaults write com.apple.finder AppleShowAllFiles -bool YES

• Show full paths in finder:

  defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES

• Note: Don’t like muck­ing around with the com­mand line, or want to have some­thing with a birds eye view that can con­trol just about any­thing, check out “Secrets” from black­tree. I’m fine with magic PLIST hacks, but this does put a nice UI on all the hacks.
• Also, if you’re a power user, or want tiling behav­ior for win­dows — see SizeUp or Divvy (I use Sizeup) both of which are excel­lent addi­tions to win­dow man­age­ment, even in Lion. I tend to be anal about win­dow coor­di­na­tion on the screen even with mis­sion control/expose.

Other than that; I’m a happy camper. The new scroll behav­ior on the track­pads is amaz­ing, if not jar­ring when using a mouse. Every­thing just pretty much kept work­ing for me. But then again, I’m pretty easy to please.

See also — “Four Lion Ter­mi­nal Hacks” from Mac­world and “Top 10 Secret Fea­tures in OSX Lion” from Life­hacker — if you really, really hate the UI changes, see “How to de-IOSify” from Life­hacker. More cool tips and tricks: “Mis­cel­la­neous Lion Tips and Tricks” and also “Mis­cel­la­neous Lion Tips and Tricks part two”.

p.s. I orig­i­nally for­got to men­tion this — but the full disk encryp­tion in Lion is imple­mented damn well — it’s easy for users to “get” — seam­less, and trans­par­ent. In my hon­est opin­ion, this is worth the price of upgrad­ing alone if you have a lap­top. It’s so well done, and it sim­ply shows how crappy fil­e­vault was — it also goes to show that if you make crypto easy and trans­par­ent for users, they might actu­ally use it. See the Ars Review for a deeper dive. See also “PGP WDE vs. Lion Disk Encryp­tion”

p.p.s. See this ArsTech­nica arti­cle about how to cre­ate a bootable copy of the Lion install disk you can use to do offline installs for other machines. I’d rec­om­mend doing the same for XCode if you have more than one mac to upgrade (or you lack FiOS).

Yes; of course I upgraded to OSX Lion on day one. To quote myself from twitter: I am happy in the warm cozi­ness and stark white­ness of Steve Jobs’ monoculture....

I hinted at this on twit­ter ear­lier — but now that I’ve had a chance to ssh into the box finally, and, well — run some bench­marks and trans­late PyPy — I am very happy to announce that the PSF has acquired quite the beefy machine for the planned speed.python.org site dis­cussed at the Python VM sum­mit at PyCon 2011.

The goal of “speed.python.org” is to port the PyPy Speed Cen­ter from “just” a PyPy bench­mark tool/suite/site and into one for PyPy, CPython, etc — essen­tially, a mas­sive source of bench­mark infor­ma­tion for a range of full Python inter­preters. These bench­marks are real world micro and macro bench­marks — some­thing we have Unladen Swal­low to thank for — quot­ing Reid:

I’m also proud of our macro-benchmark suite of real world Python appli­ca­tions which lives on and PyPy uses it for speed.pypy.org. In all the per­for­mance work I’ve done before and after Unladen, I have to say that our macro bench­mark suite was the most use­ful. Every per­for­mance change was easy to check with a before and after text snippet.

I’ll also note, some of the bench­mark suite lives in the Python HG repo — PyPy has done a lot of cus­tomiza­tions and tweaks — most of all, they’ve turned it into an awe­some site and resource, and hav­ing this data, com­par­ing say — Python 2.6/2.7/PyPy/Python3/Jython/etc with his­tor­i­cal / per build data (so we could find per­for­mance regres­sions for all of these inter­preters will be amaz­ing.

There’s already work occur­ring on the PyPy side on code­speed (the tool) to make it more generic and usable out­side of just PyPy. There’s also two Google Sum­mer Of Code projects which should help — “Turn Code­speed into a multi-project, sta­tis­ti­cally savvy appli­ca­tion” and “Devel­op­ing a bench­mark suite (for Python 3.x)”.

My part to play in all of this was to act, as the mag­i­cal PSF fairy and sprin­kle resource dust into the idea. Fun­da­men­tally, I view Python.org not as “CPython.org” — I view it as Python-the-Community.org. I view it as a resource where all inter­preters share equal foot­ing, where all of our resources should live and grow (I have a side project for that too…). This sin­gle, shared machine is step one in real­iz­ing that vision (although I’d be remiss in not not­ing that plenty of non-cpython things also share our HG repos­i­tory, and use the mail­man infra­struc­ture we have today).

That all said — I finally get to “fin­ish” play­ing the mag­i­cal PSF fairy for this, and announce, that due to an amaz­ing dona­tion to the PSF from the HP Open Source Pro­gram Office (Thanks Bob!) of a huge machine, and the gen­er­ous, free rack­ing, stack­ing host­ing and mon­i­tor­ing from the kick-ass team at OSU/OSL we now have this mega machine.

Orig­i­nally, the PSF was going to fund the pur­chase of this machine from it’s own pock­ets — but Van Lind­berg con­nected me to Bob at the HP Open Source Pro­gram Office, and bam — magic hap­pened. HP’s gen­eros­ity got us the fol­low­ing machine:

• DL380 HP DL380G7 X5670 LFF (2U)
• Dual HP NC382i Dual Port Mul­ti­func­tion Giga­bit Server Adapters
• HP Smart Array P410i/1GB FBWC Controller
• 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Reg­is­tered CAS-9 Mem­ory Kit
• 2x HP 750W Com­mon Slot Gold Hot Plug Power Sup­ply Kit
• HP iLO Advanced includ­ing 1yr 24x7 Tech­ni­cal Sup­port and Updates Elec­tronic  License
• 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enter­prise 3yr War­ranty Hard Drive
• 2   HP DL380 G7 Intel® Xeon® X5680 (3.33GHz/6-core/130W/12MB) FIO Proces­sor Kit

Yes. HP donated a box with 12 cores (24 with hyper­thread­ing enabled) and 16GB of ram, and a high speed RAID to us for this project. And man, does it fly:

Hardware:Processor: 2 x Intel Xeon X5680 @ 3.33GHz (24 Cores), Moth­er­board: HP Pro­Liant DL380 G7, Chipset: Intel 5520 I/O + ICH10, Mem­ory: 4 x 4096 MB, Disk: 1200GB LOGICAL VOLUME, Graph­ics: ATI ES1000
Software:OS: Ubuntu 11.04, Ker­nel: 2.6.38–8-server (x86_64)

OSU/OSL is going to be our host — they assem­bled the machine, racked, stacked and they’re going to be our hosts. The entire team — Lance and Peter espe­cially, have been patient, under­stand­ing and incred­i­bly help­ful. They got this machine up for us today, and helped me work through some auth issues — return­ing emails within min­utes. I’m really look­ing for­ward to con­tin­u­ing to work with them — the open source lab at ore­gon state is an astound­ing resource for open source. Astounding.

Over the next few weeks, I will be iden­ti­fy­ing the peo­ple who will be tak­ing this from my largely inca­pable hands and turn­ing it into the resource it will become — in addi­tion to run­ning bench­marks, it’s prob­a­bly going to help Brett Can­non gen­er­ate in depth code cov­er­age reports for the stan­dard library — some­thing we intend on break­ing out and shar­ing amongst the inter­preters as a com­mon code base.

This is great news — and I’m look­ing for­ward to the next few months as the hard part of the project takes off — get­ting the speed site up and run­ning and work­ing for the com­mu­nity. For the record, I had to test the speed of this beast some­how — so I picked the most resources inten­sive thing I could think of — trans­lat­ing PyPy using CPython 2.7:

[Timer] Timings:
[Timer] annotate                       ---  732.2 s
[Timer] rtype_lltype                   ---  482.5 s
[Timer] pyjitpl_lltype                 ---  533.4 s
[Timer] backendopt_lltype              ---  146.4 s
[Timer] stackcheckinsertion_lltype     ---   38.9 s
[Timer] database_c                     ---  184.9 s
[Timer] source_c                       ---  344.7 s
[Timer] compile_c                      ---  137.5 s
[Timer] ===========================================
[Timer] Total:                         --- 2600.4 s

As a point of com­par­i­son — check­ing the lat­est build bot result from PyPy:

[Timer] Timings:
[Timer] annotate                       --- 1452.8 s
[Timer] rtype_lltype                   ---  748.6 s
[Timer] pyjitpl_lltype                 --- 1206.6 s
[Timer] backendopt_lltype              ---  560.5 s
[Timer] stackcheckinsertion_lltype     ---  109.9 s
[Timer] database_c                     ---  712.8 s
[Timer] source_c                       --- 1295.1 s
[Timer] compile_c                      ---  472.6 s
[Timer] ===========================================
[Timer] Total:                         --- 6558.8 s

So, yeah — not apples to apples, but there’s a slight speedup.

We now have the foun­da­tion for an amaz­ing, real world bench mark and test­ing site for Python/Python’s inter­preters. This is going to be awe­some, and it could not have been done with­out the work and dis­cus­sion by the peo­ple at the PyCon lan­guage sum­mit, the PyPy folks for putting together an awe­some site, and with­out the sup­port of the PSF — but most of all, the gen­er­ous sup­port from HP and OSU/OSL. My thanks goes out to every­one involved, and I’m hon­ored to be able to help get this project off the ground finally. Let’s show every­one what Python — as a com­mu­nity, work­ing on con­cert with one another is capa­ble of doing.

PS: Here’s a run of the basic tests from the cpython repo bench­mark tool

I hinted at this on twit­ter ear­lier — but now that I’ve had a chance to ssh into the box finally, and, well — run some bench­marks and trans­late PyPy...

(photo cour­tesy of Sean MacEn­tee via flickr)

Some inter­est­ing bits and pieces (left­overs) from the week­end — I have a ten­dency to queue up a pile of “read later” stuff or email­ing myself a pile of things to talk about/link to/etc. Some­times, I actu­ally get to go through it all. Today — I get to share it to!

First up — Michael Foord (fuzzy­man) did two excel­lent posts — the first is on pri­vacy in Python — not big-P pri­vacy, but rather the programming/object pri­vacy. It’s a good read because it reen­forces the point that even if you think you’re being clever and hid­ing some­thing in a clo­sure to make it pri­vate, you can still get access to it. Remem­ber too — dun­ders (__foo) are just name-mangling. In Python, noth­ing is pri­vate (insert bad tast­ing joke about Python being face­book here).

Michael’s sec­ond post is on the Named­Tu­ple ker­fluffle that was stirred up Krist­jan Valur’s post on the use of exec() and named­tu­ple (short ver­sion: named­tu­ple cre­ates a string defin­ing the class and then calls exec to cre­ate the object. I think Michael is spot on — I think exec() gets a bad rap frankly, sure — it’s some­thing of a hand-grenade, if you use it wrong, you’re going to get hurt, but in this case I have to agree with Ray­mond in his com­ment on bug 3974 — the cur­rent imple­men­ta­tion is clear and main­tain­able. I don’t like the stern­ness of the reply — but he is right. Kristjan’s use-case is an inter­est­ing one, but I don’t think it’s one that war­rants the removal of the exist­ing imple­men­ta­tion. I’m won­der­ing if a “fall­back if exec doesn’t exist” is worth inclusion.

Then again, I’ve spo­ken to peo­ple who refuse to use named­tu­ples because they now know how the sausage is made. I still think the sausage is deli­cious.

Next is a pretty inter­est­ing dis­cus­sion on a pre­sen­ta­tion that came out of Pixar on get­ting over embar­rass­ment in order to get things done — I don’t have much to add above the com­ments on hacker news, except to say I think the same men­tal blocks they talk about for ani­ma­tors apply to pro­gram­mers, writ­ers, etc. We hide from code reviews, we hide our writ­ing until we think it’s “Pitch Per­fect” — when in real­ity, we shouldn’t. We should be shar­ing, dis­cussing and col­lab­o­rat­ing ear­lier, more fre­quently and more often.

Share and Ship early and often — see also “Don’t Go Dark”.

Finally, I was happy to see the “Quick and Easy Debug­ging in Python” post from Jeet Suku­maran — it’s always nice to evolve past sprin­kling print fairy dust all over your code for debug­ging — even if we all still do it despite know­ing or using PDB. Just to add to his post, if you want to increase your PDB-Fu, I rec­om­mend this “Debug­ging in Python” post by Steve Ferg, and the offi­cial doc­u­men­ta­tion for PDB. We should really add a HOWTO for this.

p.s. For addi­tional good-reading, check out “Pri­or­i­ties Don’t Exist in a Vac­uum” and “First Care”- while not ger­mane to what’s I’ve already writ­ten about, they’re a good essays on pri­or­i­ties. Thanks to the lat­est “Back to Work” pod­cast.

(photo cour­tesy of Sean MacEn­tee via flickr) Some inter­est­ing bits and pieces (left­overs) from the week­end — I have a ten­dency to queue up a pile of “read later” stuff...