You can see from the post quite a bit of the capital spent goes to support other conferences — as I’ve stated before, money that comes into the foundation in the forms of donations and PyCon “revenue” goes back into the system to be issued out to things like this.

This is why I am so hot to encourage grants around Porting to Python 3 — I think that the PSF can, in the next year, increase grant work for conference and outreach as well as developer work (such as porting libraries and other projects). None of these things should be solely focused on CPython alone — PyPy, Jython, etc should all be recipients of grants.

And therein lies the rub.

The PSF does not “go looking” for places to issue grants — the PyPy grant at PyCon 2011 was a bit of an aberration in that I proposed it to the board directly.

We need applications from the community! We can do things such as cover meetup fees for user groups, or help fund conferences, or development work. Jessica McKellar, I and others recently revamped the PSF grants page to hopefully provide a better outline of how grants work.

If you have more questions — feel free to ask me here or via email — the PSF’s mission is happily broad, and we’re here to serve and represent the community as best we can. But we do need to hear from you!

In this post’s case, I’m taking stock of the things that changed for me — things that stick out in my mind and projects I’ve either started, floundered or run completely into ground.

Design and Experience Matter

Perhaps the biggest shift for me in Python-as-a-whole is a movement more towards the social / management aspects. I’m a Python Software Foundation board member, so obviously me needing to take a “bigger view” isn’t that surprising. What has been surprising to me is that everywhere I turn, I see things we as a whole can do better.

Now, before you think I’m about to go off the deep end; let me assure you — I wouldn’t trade the community I’m lucky to be part of for anything, as I’ve said more eloquently before. However, only a fool believes that anything is perfect, and only the insane only focus on the flaws.

Taking a step back, I’ve seen more and more things that I think we can do a better job at, and these realizations all revolve around my continued “transition” from more back-end to more front-end design and coding. As I’ve become more focused on the users/community and those who are new, I’ve grown to internalize the fact that design and experience matter not only in code, and in a GUI, but they matter to a community and language as a whole.

I’ve spent the better part of this past year focused on issues around this — encouraging people to get involved in the “softer” side of things — helping out with documentation, mentorship and education, trying to get people to think more about one another and those just getting started and introduced to things.

I think that we as a community — and I mean everyone — from Django to Plone, from Twisted to Tornado, from PyPy to cPython can take a look at the “more human” aspects and find things to improve. Sometimes it requires fresh eyes to show you what’s broken — people who do code reviews regularly know this.

For an example, look at Kenneth Reitz’ Requests module — billed as “HTTP for Humans” — this might be a perfect example of the point I’m trying to get across. Built on top of “less friendly” libraries, it’s API is a joy to use. It’s simple, it’s clear — the documentation is well done and the entire project feels very welcoming. Perhaps “Welcoming” is the best word for what I’m looking for.

I get stuck in wanting to fix “all the things” — and I can’t help but get mired down in the details of how we make everything more welcoming and the experience better, how do we lower the barrier and reduce friction. The result is that I’ve broken my promises to myself and taken on more things than I can possibly hope to do justice.

How do we make things more welcoming, how do we help the new people, how do we help those of us growing stuck in our ways to find and explore new things? How can we do this as a community to lift us all up? What I think we need is a series of small, positive changes. Little things like, say:

• User friendly READMEs and Documentation. Yes — I said friendly — don’t assume your users are magical super smart engineers and users. While the article is more web focused, I enjoyed “The Myth of the Sophisticated User” — please don’t assume people are running bleeding edge version of everything, and please don’t assume everyone knows 20 years of Python package development.
• Mentorship! Set up something within your project or team that is focused on mentoring people to a point where that person is comfortable to be a contributor.
• Stop the vitriol. If you find yourself angry when you’re typing that reply to a mailing list; walk away. If you see others being hostile or just flat out rude, call them out on it (privately first, no reason to be a jerk). Aim to be polite and welcoming.
• The next time you’re putting something up on the web? Take a moment to think about or learn about making something — yes — pretty and usable. Even if it’s something simple, take a moment to realize that you’re building something that may be your future user’s first experience with you. It may be as simple as picking up “Design for Hackers” (which I quite liked) or just going with something with sane defaults — like twitter bootstrap.
• Speaking of sane defaults — please be opinionated. When a new user wants to install something, don’t give them the complete history of packaging, just gently explain to them how to do it. Even if I don’t agree with the way you do that, it’s a far cry from 20 years of development history being dumped on someone when a simple pip install <blah> could work. The same goes for your software: Pick sane, rational defaults and abstract away as much as you can. Put examples of usage before the API in documentation.
• APIs and syntax matter: your communications channels to your users are APIs and syntax just as much as your actual code and libraries.

Moving on — I hate to say it this way; but think of the Users and target audience. Remember, you — the person reading this — and I — are in a tiny minority of the population where software (for the most part) isn’t magic, we understand history and we’re very tolerant of unfriendly things and failures because that’s how we “grew up”.

Not everyone knows how to build an interpreter; or a web framework — it doesn’t mean they still can’t contribute.

The Python Software Foundation

As most of you know — I am one of the directors of the Python Software Foundation, and have been the past two years. 2011 was another year where the PSF got to do some pretty cool things. I’ve been stressing and pushing more and more that the PSF has to be focused not just on the “IP” of Python, or just on cPython development — we have to take a larger view of the entire community — this means encouraging projects such as PyPy, outreach workshops, conferences, etc via grants and support.

You should really take a look at the Python Software Foundation’s blog — Doug Hellmann, Brian Curtin and others have done their best to document and showcase what the PSF has been up to, and where we’re trying to help.

My primary focus has been encouraging things such as the Outreach and Education committee, and working behind the scenes with a lot of people to improve the Python.org infrastructure. More recently I’ve been working on a project which should hopefully become public soon — but is tied to my first point about Design and Experience and the PSF.

I want the PSF to grow in the good works it performs — more grants as we can afford it, getting better hosting for things as needed, helping out projects like Read The Docs or helping push forward Python 3. The PSF is the Python Software Foundation — we need and should be supporting and helping everything from PyPy to PyPI, cPython to Scipy.

I think the best way for me to help here is to pick up where I left off documenting the PSF. Once again — the design and interface matter.

The Sprints Committee

As part of my board work back in 2010 I helped start the Python Sprints project — and under Brian Curtin’s guidance in 2011, it has continued to make small donations in places it matters. In 2012, I’d like to see if I can spin back around and help it grow more and flourish, perhaps even be able to provide more money where it’s needed. It’s growth has been slow — but that’s also due to us seeing less sprints overall it seems.

GetPython3.com

Started as a side project (yes. another one. sigh.) Get Python 3 is meant to serve as a pile of information and resources about Python 3 — and as many of the aspects of Python 3 as possible. Where to get funding, how to port, what is ported. I’ve actually gotten some excellent help from others (see github) and I’m hoping to grow it more. I’ve gotten pretty good feedback on it — and I never turn down a patch!

Python (Core) Mentorship

Driven from my experience with the first point about being welcoming, I’ve done my best to spin up the Python Core Mentorship group, a team / list focused on mentoring new people into contributing to core Python. To quote the home page:

The mission of the Python Core Mentor Program is to provide an open and welcoming place to connect students, programmers – and anyone interested in contributing to the Python Core development. This project is based on the idea that the best way to welcome new people into any project is a venue which connects them to a variety of mentors who can assist in guiding them through the contribution process, including discussions on lists such as python-dev, and python-ideas, the bug tracker, mercurial questions, code reviews, etc.

While traffic is low, I think it has done it’s job — as with everything else on my list, I’d like to see growth — as it is, due to everything else on my plate, others have stepped up to help lead and guide the group. As it is, I’ve run into a case where as I’ve found with many other projects like this — people are already “tapped out” — myself included. More on resource contention later — and I should really do a poll and gauge the list for the relative level of success they feel the group has engendered.

Python Speed Project

Another side-burner project is the Speed.python.org project — this one makes me sad(der) than my other time-starved projects. While we have finally been able to set it up as a PyPy build slave and have it feeding results to speed.pypy.org (see the speed-python results), it has not taken off as much as I hoped. We have a beast of a machine (see my initial announcement) — but we’ve hit the resource wall like everything else. Not enough people with enough time and the right skills.

The Elephant in the room: PyCon 2012

My single biggest project this year has been getting PyCon 2012 ready to fly — everything from getting the new website launched, the staff assembled, writing a code of conduct, and providing white-glove service and support (and getting) our amazing list of sponsors.

I can’t really estimate how many hours I’ve “worked” on Python — but I can tell you every hour has been worth it. Even though it’s sucked my time from other things and projects, it looks like it’s going to be an amazing conference. We have robots, we have amazing talks, amazing keynote and plenary speakers (Paul Graham and Stormy Peters for starters). We have awesome tutorials and even more to come.

PyCon represents the single biggest “community act” that the Python Software Foundation performs — not only does the PSF fund PyCon, but it manages it, assumes the risk, etc. I wrote about it in detail in my post “Making the Case for Sponsorship” and in the “Everybody Pays” post. I’m hoping to continue to write up more and more of the details of the inner workings of PyCon, as I think it’s an important series of data points and lessons. Remember — any funds “left” from PyCon go the PSF which allow the foundation to issue grants to other conferences, to developers, groups and workshops. It helps us help you.

PyCon 2012 is the thing I am most proud of; we have 80 sponsors and partners (Such as OpenHatch and PyLadies), we have a solid team of organizers working together to bring PyCon 2012 to fruition. We have a robust financial aid program as is tradition. I can only hope that I have the tenacity and will to see it come together and be able to look at a sea of 1500 Pythonistas — new and old in Santa Clara.

ps: You can register here. :)

Blood from a Stone

How do you get more time from people who are busy? Time and Time again, I’ve found myself asking that question. Each one of the projects I’ve listed has hit the same issue over and over again. How do you get the volunteers necessary to help? Heck, even my call for help with multiprocessing in August fell on a mostly flat note — probably due to me.

I no longer feel “ok” asking for help with new projects simply due to the fact that I know everyone is busy — it’s insane of me to ask people to take their time away from their projects or families or jobs.

What that means however is that I have completely failed in the not-taking-on-new-things department — and I don’t see this changing much without me flat out learning to tell myself “no”. I believe in this community — I believe in the people, the friends I have, the language and everything involved. It’s not just another tool for me; it never has been. I’m still learning, and mostly failing (or flailing, depends on where I’m standing).

Finishing this one off

Looking at the list I’ve typed out above, I suddenly have the feeling that I didn’t actually do much last year, I know thats wrong (a nasty look from my family members would easily remind me of that). I have been able to help out where I can making things more friendly, more welcoming and to reach out when and where I can to offer help, and support.

I’ve watched the community change in some dramatic ways, I’ve looked on as PyPy has gained amazing momentum, more and more vendors and companies have come out with Python support and stating that they’re using Python (and are hiring). I’ve gotten to work with PSF members, the board, and many, many others — all I can do is keep at it, and hope I do things justice.

Here is a working example (forms.py) — just so I don’t forget it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from django import forms
from django.contrib.auth.models import User
from django.contrib.auth.forms import UserCreationForm
 
class UserCreateForm(UserCreationForm):
    email = forms.EmailField(required=True)
 
    class Meta:
        model = User
        fields = ("username", "email", "password1", "password2")
 
    def save(self, commit=True):
        user = super(UserCreateForm, self).save(commit=False)
        user.email = self.cleaned_data["email"]
        if commit:
            user.save()
        return user

You have to modify the save method on the form to add the email to user object returned by the super call. You can use this to expose other fields on the User object as needed.

Recent posts and discussions around porting of existing libraries and frameworks to Python 3 have been pretty interesting. I think that there have been a lot of good points brought up in the discussion (See: Armin’s Post (and followup), Nick’s entry on Python 3 and Nick’s email to Python-Ideas).

On a personal level; I’ve felt frustrated that there’s not much that I can do myself — I do believe that 2.7 is the proper end of the road of Python 2, and I do think that Python 3 is the future of the language. Does that mean Python 3 is perfect? Oh hell no. Does it mean that we can do work to make Python 3 the “Python 3″ we all want and need?

Yes it does.

So; while there is nothing I can do directly other than continue to work on the site I’ve been slowly building — GetPython3.com with help from the community — there is an aspect I can help with from a Python Software Foundation / Grants level. That means money (well, not unlimited).

As some of you might know — the PSF has actually issued grants to developers who have applied to port important libraries to Python 3 — as I say on the GetPython3 page:

In short: yes — there’s a bevy of information, videos and blog posts out there that can help you on your way. Python 3 is the future of the Python language, and entities such as the Python Software Foundation strongly believe in supporting the porting effort.

For example, the Python Software Foundation has issued developer grants to port projects such as the email package, PyOpenSSL, and WebOb. It has also provided developer grants for other general Python development work, such as to Brett Cannon that allowed him to completely revamp the Python developer’s guide.

The Python Software Foundation is here for not just CPython, or python-core, or python-the-language. It is here for Python — the community, it’s efforts, its developers, designers and people.

Certain projects — most notably PyPy — have already started donation programs to help fund large-scale development efforts to Python 3. Others may soon follow.

Additionally to the grants-to-developers aspect — the PSF Sprints project has been issuing grants for Python sprints in general, which means you can apply / ask for a grant for a port-to-python3 workshop or sprint any time!

But; back to where I was going…

My offer to you, the community is this — I can not guarantee you will get a grant, or funding — but what I can do, and what is within my power as a fellow member and PSF Director is offer to help write, and review applications to the PSF Board of directors for grant applications.

That’s right — I will assist you in writing an application that will be submitted to the PSF Board for approval, for grants aimed at porting libraries or frameworks to Python 3; or doing specific documentation / core work for Python 3. I can help you write it; provide templates, discuss it with you (I may have some elves help me) and ultimately help you put it in front of the board for approval.

Obviously; the PSF does not have unlimited funds; nor can it spend funds irrationally. Python 3 is important however — critically so — and while we can not fund everything, we can do what we can. I am aiming at libraries/frameworks which are in widespread use (e.g. notable) and that other projects/libraries/frameworks depend on heavily (for example, see the Py3k poll).

Before getting started, you should read the basic PSF Grant guidelines and you should look through the information on http://getpython3.com/.

If you are interested in this; drop an email to jnoller@python.org — I don’t promise immediate up-to-the-second turn around — I’ve obviously got a lot on my plate right now, but I will do my best to help.

Whether you’re a first-timer or an experienced veteran, PyCon is depends on you, the community, coming together to build the best conference schedule possible — PyCon is first and foremost about the community, driven by volunteers both on an organizational level, and by speakers.

Our call for proposals lays out the details it takes to be included in the lineup for the conference in Santa Clara, CA on March 7–15, 2012. I should note that this year we have gone away from consistent groups of invited speakers — meaning, all talks, regardless of who submits them are not guaranteed a speaking slot. All talks are judged and reviewed on the merits of the talk and the speakers themselves.

If you’re unsure of what to write about, our recent survey yielded a large list of potential talk topics, and plenty of ideas for tutorials. We’ve also come up with general tips on proposal writing to ensure everyone has the most complete proposal when it comes time for review. As always, the program committee wants to put together an incredible conference, so they’ll be working with submitters to fine tune proposal details and help you produce the best submissions. Even if you are still incubating a talk idea: submit the proposal now in rough form and we can assist you in fleshing out and refining the proposal during the review process.

We’ve had plenty of great news to share since we first announced the call for proposals. Paul Graham of Y Combinator was recently announced as a keynote speaker, making his return after a 2003 keynote. David Beazley, famous for his mind-blowing talks on CPython’s Global Interpreter Lock, was added to the plenary talk series.

Sponsors can now list their job openings on the “Job Fair” section of the PyCon site as we previously announced — providing an excellent resource for job seekers, and providers.

We’re hard at work to bring you the best conference yet, so stay tuned to PyCon news at the PyCon blog and on Twitter at https://twitter.com/#!/pycon.

We recently eclipsed last year’s sponsorship count of 40 and are currently at a record 54 organizations supporting PyCon. If you or your organization are interested in sponsoring PyCon, we’d love to hear from you, so check out our sponsorship page.

And as always — quick thanks to all of our awesome PyCon 2012 Sponsors:

• Diamond Level: Google and Dropbox.
• Platinum Level: New Relic, SurveyMonkey, Microsoft, Eventbrite, Nasuni and Gondor.io
• Gold Level: Walt Disney Animation Studios, CCP Games, Linode, Enthought, Canonical, Dotcloud, Loggly, Revsys, ZeOmega, Bitly,ActiveState, JetBrains, Caktus, Disqus, Spotify, Snoball, Evite, PlaidCloud and Mozilla
• Silver Level: Imaginary Landscape, WiserTogether, Net-ng, Olark, AG Interactive, Bitbucket, Open Bastion, 10Gen, gocept, Lex Machina,fwix, github, toast driven, Aarki, Threadless, Cox Media, myYearBook, Accense Technology, Wingware, FreshBooks, and BigDoor
• Lanyard: Dreamhost
• Sprints: Reddit
• FLOSS: OSU/OSL, OpenHatch

Thank you — and as always, feel free to reach out to the team or any of the staff with any questions you might have.

• The PyCon Organizers — http://us.pycon.org/2012
• Jesse Noller — Chairman — jnoller@python.org
• Brian Curtin — Publicity Coordinator — brian@python.org

I wanted to take a moment to explain what makes sponsorship good for the community, and a sound investment for sponsors new and old, prospective and future.

This year, as chair, I’ve taken it upon myself to push and manage PyCon sponsorship (corporate, non profit, media, etc) for a variety of reasons. First, as someone who has been a sponsor in the past (and present) and as someone who spends a lot of time “selling” the Python Software Foundation, and the community to others — I feel very closely tied to PyCon and sponsorship.

Not to mention — corporate sponsorship is what allows us to keep this probably one of the least expensive international technical conferences you could possibly attend this upcoming year. Without sponsorship — and the array of sponsors we have right now for PyCon 2012, the conference could simply not happen at the size it has reached, or have a robust financial aid program, keep tickets and tutorials cheap, etc. We have, once again intentionally capped attendance at a level to allow for this, and to help keep PyCon’s community feel and closeness.

Running a conference is, frankly, a dangerous game. As I noted in my blog post several months ago discussing some of the financial workings of PyCon and its financial philosophy. It is very easy to lose a lot of money, very quickly. PyCon is held / financed / backed by the Python Software Foundation. This means lack of sponsorship, low attendance, etc could — with a simple misstep — bankrupt the foundation. Sponsorship helps shore up the gamble you make signing contracts on catering, room bookings, rental of the space where the conference is held, audio/video costs, etc. Although, if you make a big enough mistake — nothing will prevent things from going south. This means careful planning, budgeting and negotiation.

Also, while PyCon has always been, and will continue to be a community focused and therefore, low cost and inclusive conference, not really focused on profiting from attendees, any revenue that comes out of PyCon (profit, if you will) goes directly to the Python Software Foundation. This money, in turn, is used to improve infrastructure of Python resources, provide developer grants for programming work, provide grants to conferences all over the world and many other community projects.

In the last few months alone the PSF has issued grants to PyTexas, EuroPython, Python Ireland, PyCon India, and many, many others. We have issued grants for porting modules to Python 3, service such as Read The Docs, etc. Any revenue/profit is flipped back into funding PyCon, and the community as a whole.

PyCon provides a very tangible entity for corporate sponsors — it’s an easier “sell” than direct PSF sponsorship, and therefore is a fundamentally better conduit for funds into the PSF.

That’s all fine you say: those are great things for the community, and conference — but why would a company want to sponsor PyCon? Sponsors receive tangible benefits such as recruiting at the conference, advertising and marketing, getting community involvement known (call it community karma), etc. Sponsorship isn’t just a matter of asking a company to fund the conference because “it’s good for the community” — it’s a matter of showing them that not only is it good for the community — it’s good for their goals and needs.

PyCon is an excellent recruitment tool.

If you’re looking for Python programmers, a venue filled with 1500 Python hackers of all types — from web developers, to designers, to distributed systems engineers and operations people is an excellent place for you and your company to find “that special someone”. I know a lot of Python hackers out there who have been hired by companies they “met” at PyCon. I also know a lot of speakers and tutorial teachers who have received jobs or job offers after speaking/teaching at PyCon.

Just as PyCon is an excellent venue for companies looking to hire, robust sponsorship allows people at the conference know what companies out there could be hiring Python hackers. Companies like Walt Disney Animation Studios, Google, Dropbox, and others as well as companies that aren’t well known for being Python shops. It’s a great venue for job seekers to find employers.

The Jobs Fair page we added this year for sponsors, and those looking for jobs is a logical extension of this. Anything we can do to connect people and companies is great.

PyCon is an excellent marketing tool.

If you are looking to sell something — an editor, hosting, a service, etc — PyCon’s 1500 attendee pool provides an amazing cross section of people. Not just hard core developers — entrepreneurs and startup founders, IT business people and leaders. Python is a language that as time goes by — I am less and less surprised where it pops up — and more surprised when it isn’t being used somewhere within a company.

It is literally everywhere — a frequently unsung hero for many companies. Sometimes, companies use it without even knowing it.

Python — and it’s community — and therefore PyCon is amazingly diverse. This means when you sponsor PyCon, you are advertising to an amazingly diverse group of people. Skill sets from all walks of technology — and a surprising number of people to whom Python is a tool they use prolifically to get some other job done (like say, video rendering or controlling robots). PyCon’s attendees reflect the stunning makeup of it’s community. You can’t go wrong getting your companies names on attendee’s lips.

PyCon is a great way to raise visibility.

This is as much a sub-point of my previous note on marketing as anything else but it deserves some attention. If you’re a company who is trying to get the word out, trying to spread the news about your new product or service, people notice PyCon sponsors. Not only are you listed on the website, you get signs, booths and entries in the program guide at the conference. It can be en excellent tool for buzz and discussion about and launching a new product or service.

Even if you’re not selling something — and you just want to get the word out about your company’s open source efforts, opinion and ideas and use of Python — PyCon is a fantastic platform to do so. It can literally be a platform you use to launch you name and brand into the community’s shared mind.

PyCon sponsorship breeds good will.

I wish I had studies to show it, but people within the community and at the conference itself see companies sponsoring PyCon and understand that while those companies might be selling, marketing or recruiting — are still doing the community a huge favor by acting as sponsors. As I said before — the community benefits are many, just as the sponsor benefits are. I can not stress this point enough — the companies that help PyCon via sponsorship or attendance do it for many reasons — some of them financial, but the social aspects are something all of our sponsor from the past can attest to. Python is an open source language, with a strong open source ethos running through its community — and seeing companies give back both through code and financially means a lot to everyone in the community — even other sponsors.

PyCon sponsors help set an example for the community in terms of involvement and support.

PyCon sponsorship is a good, simple and cost-effective investment.

I wish all conference had sponsorship packages as cheap and as robust as the ones PyCon has outlined in it’s prospectus. Heck — a good recruiter to find talent can cost a company $30,000 or more alone — by comparison, the sponsorship levels and prices PyCon has are fantastic deals (especially when you factor in that companies under 25 people can get a 50% off discount on two of those levels). For less than a price of a good computer and monitor — you can be a Silver sponsor. For less than the price if you include the desk and furniture or software licenses? A Gold sponsor. For less than the price of a good recruiter, or Google Ad campaign? You can be a Platinum or Diamond sponsor and reach out to not just PyCon attendees but to the entire Python community.

PyCon is a professional event.

I swell with pride standing in the shoes of the conference chairs that have come before me. PyCon, while focused on the community, the language, learning, teaching, being a ton of fun for all of its attendees, and excellent location to hack and network is one of the most friendly-yet-professional conferences I have ever had the privilege to attend.

PyCon is backed by the Python Software Foundation — but it is run by volunteers — even I, as chair, am not paid. For all of us involved, it’s a labor of love. It is a way for us to give back to the community, ecosystem and companies and sponsors attending or sponsoring. And while it may be volunteer based — it’s 100% professional. From the website, to the program guide, from talk selection and booth assignment — everything is treated with sincerity, respect and trust.

Sponsors can look at PyCon not just as a good investment, or platform — but as a safe one — and if they can not, I have failed as chair of the conference. The same applies to every single attendee.

But too much of a good thing?

As with all things, there is a flip side to this. Sponsorship is great for sponsors, and the community — but PyCon is fundamentally community focused, and hence we must walk a line between having robust sponsorship packages, and going the “full sponsorship monty” so to speak. This means that to this day, I hold firm on the policy that sponsorship does not guarantee or provide tutorial or speaking slots to any sponsor.

At PyCon, we are all equals, especially when it comes to talks. Joe developer from nowhere, Antarctica can submit a talk, tutorial or poster session as can Bob the developer from a Diamond sponsor and they have equal chances of being accepted. If the talk is good, if the speaker is known to be a good speaker, if the content and subject are compelling, a proposal will be accepted on its merits (but even then we can not accept all the deserving ones).

Other conferences guarantee speaking slots for sponsors — I feel this runs counter to the PyCon ethos and community philosophy. Not only are we open in our source, we treat each other as equals and with respect. Ours is the meritocracy of ideas and work — and this point can not get lost or forgotten in our — my — work on our sponsors’ behalf to increase the value and return on investment they see.

We also try to keep the advertising and visibility at the conference tasteful — limiting banner sizes and locations, focusing on the vendor area experience while also giving sponsors free admissions to the entire conference so they too can partake in the learning, hacking and networking. We find this to be a good balance between the needs of the attendees and the needs and desires of the sponsors.

Trust me, if I thought walking around in a NASCAR-like track suit covered in logos would help our sponsors, I just might — ask the other staff! But that’s just me.

In closing — I want to encourage you and companies you know or work for, to take part in PyCon and get involved. Even if you can not, or do not want to be sponsors, I encourage you to submit proposals, lightning talks when the conference comes, attend the sprints, and recruit on the “down-low” by just talking and hacking with everyone.

I encourage you, and will work with you day and night to join us as sponsors — but I value your involvement in the community, and the conference more. Even by just attending, you are enriching us all. If you have suggestions on how to make sponsorship better for sponsors — or general comments or concerns, feel free to email me.

Giving thanks

Finally, I’d like to thank all of our current sponsors — and an a yet-to-be-named mystery sponsor:

• Diamond Level: Google and Dropbox.
• Platinum Level: New Relic, SurveyMonkey, Microsoft, Eventbrite, Nasuni and Gondor.io
• Gold Level: Walt Disney Animation Studios, CCP Games, Linode, Enthought, Canonical, Dotcloud, Loggly, Revsys, ZeOmega, Bitly, ActiveState, JetBrains, Caktus, Disqus, Spotify, Snoball, Evite, and PlaidCloud
• Silver Level: Imaginary Landscape, WiserTogether, Net-ng, Olark, AG Interactive, Bitbucket, Open Bastion, 10Gen, gocept, Lex Machina, fwix, github, toast driven, Aarki, Threadless, Cox Media, myYearBook, Accense Technology, Wingware, FreshBooks, and BigDoor
• Lanyard: Dreamhost
• Sprints: Reddit
• FLOSS: OSU/OSL, OpenHatch

And of course, if you want more information on sponsorship — visit the PyCon 2012 Sponsorship page.

Yesterday, I made a statement on twitter:

I am officially looking for someone to take over multiprocessing maintenance from me. http://bugs.python.org/issue6721

Ignoring any comments in that bug; I maintain that a later tweet is still true:

Sometimes good points and poignant criticism can be buried in a pile of crap.

In hindsight; I could have worded the original message differently “taking over maintenance” means that I am, and always have been the sole contributor to the multiprocessing code base, which is patently false. Antoine, and many other python core developers, and people within the community have submitted bug reports, patches, tests and documentation. My words were intentionally harsh — but the direction of that harshness was to me; I feel that as the “leader” (for some measurement of “lead”) I have been remiss in my responsibilities and leadership.

Sure; I could be less harsh on myself — but the level of expectational debt that I’ve incurred against myself for the module and the maintenance has grown, and grown. Even if I find myself leading PyCon, busy as a PSF Director, pushing the core-mentors program, the sprints program, and a lot of other community projects, I am still responsible for the care and feeding for the creature I helped create and birth. I’ve committed the sin of “going dark”.

For some history, see:

• Multiprocessing in Hindsight
• Python 3.0, some multiprocessing info, administrative notes.
• The 2.6/3.0 Multiprocessing todo list
• Programmer Insecurity and Mea Culpa

Months ago — I spun up the multiprocessing-sig mailing list, in hopes to engage more people — highly active users, interested people, etc to help me pay down the debt. Of course, in retrospect; it’s unfair for me to expect anyone but me to help me pay down the debt I’ve incurred. On the other hand, the responsible thing for me to do — the mature thing for me to do — is to ask for help — not to “wash my hands” of anything, but rather to take this as an opportunity to look as multiprocessing as something greater than what I originally envisioned and submitted to core.

I hear from people every day who are using the module — every day, something I helped birth helps people get things done. Multiprocessing has grown up by virtue of becoming part of Python core, and daily — despite the bugs, the debt and the quirks — it helps developers achieve something they might have otherwise been unable (or at least, had a more difficult time) to do.

The module is expansive — it has pools, tools for distributed programming via managers, pipes for interprocess communications, it’s feature set is both large, and ultimately complex in its underpinnings. That complexity — that feature set — is the reason why that debt, the bugs, the quirks has grown over time. If it wasn’t being used — I wouldn’t have so many emails about it — or bugs filed against it.

So where are we/it today?

Today, multiprocessing has widespread usage — in Python 3, there’s actually a new module named concurrent.futures that builds on the building blocks of multiprocessing and threading. Packages like Celery use it extensively (and work around internal quirks). For Python 3 — the sky is the future for what multiprocessing could be — additional functionality, moving parts of it (such as the pool abstractions) into the concurrent namespace, extending and improving the Manager classes, etc. For Python 2.7 — bug fixes, doc fixes only.

If you search the Python bug tracker for the word “multiprocessing” regardless of assignee, you’ll get 119 hits. That’s right; 119 — not all of them are multiprocessing bugs — and many of them are dupes, or fixed in recent versions. What that query gets you is an idea of the debt that has to be paid down and resolved. Each one of those bugs needs to be looked at, reproduced, de-duped and patched. Some of them may be documentation issues, some are pretty hairy (like the aforementioned http://bugs.python.org/issue6721 as well as http://bugs.python.org/issue4106 and http://bugs.python.org/issue8713).

What I asking for — rather than washing my hands of anything, or any attempt to absolve myself of responsibility, is for help. I am stretched thin — too thin to do this myself, or to be the only person who can maintain, understand or work on this module. It’s too big for that, it’s too important for me to be the arbiter of it any longer. It’s bigger than me.

Aside from the bug queue; there’s a short list of things that need to be done — the docs need to have a fresh, hard set of eyes on them, there are things (behaviors, features) that are undocumented. The test suite needs a complete overhaul — when I inherited the code, this is the first thing I should have done — but I didn’t. The problem is that the test suite is mired in magic and complexity, and without an expansive, maintainable test suite, I don’t feel confident that the bug list can be addressed with confidence.

So, I come to you with my hat in my hands, a humbled man. It’s unreasonable for me to ask for others to “pay down” the debt I’ve incurred; but it’s irresponsible, immature and misguided of me to think that I alone, or any single person can go at this alone. So I need your help — and, if in time, someone choses to be the “leader” for the module, then I will gladly step back. Until then, I will try to continue to be a guiding hand and at least point people in the proper direction, commit patches, etc.

If you are interested; please speak up — or just wade into the bug queue. You can sign up to the multiprocessing-sig list, and ask questions there, or if you’re new to core Python, and want some additional mentorship, check out the Python Core Mentorship program — that list serves as a gentle and polite, welcoming introduction to core development. No question — no matter how green — is off limits, and it’s already got an excellent track record of helping people get up to speed.

In closing; I’m going to apologize — we all know lives change, careers change, and interests change. All of these things have happened to me, but in changing so quickly and taking on different roles, I left something important behind. In doing so, I have done a great disservice to you, the community and users.

I will also thank you; without you — the users, current and future helpers, multiprocessing wouldn’t exist or be relevant in any context. Without you, I wouldn’t have the drive to even write this post, fight to get multiprocessing into the std lib to begin with, or perform any of the other roles I do.

So; thank you.

Yes; of course I upgraded to OSX Lion on day one. To quote myself from twitter:

I am happy in the warm coziness and stark whiteness of Steve Jobs’ monoculture.

Regardless of that though, I had very few hiccups with Lion itself — but a few things you need to deal with going in:

• Be on a high bandwidth connection. The downloads you have to make are huge.
• XCode 4.x is now free in the App Store — you need this — the first thing you need to install after Lion is the latest version of XCode from the App Store, if you do this, all your virtualenvs, your homebrew environment (at least mine) just Keep Working. Save yourself some pain.
  • Side note: @kennethreitz, gentleman scholar, has actually done a custom osx-gcc-installer — this contains a system install of GCC and all the goodness you need (such as install_name_tool) for Python hacking. So you might be able to skip the massive XCode install.
  • When installing XCode, for some unknown unholy reason, if you have not quit itunes, and itunes helper (see activity monitor) prior to starting the XCode installer, the install will hang. Do yourself a favor and kill it with fire.
  • Remember; the binary directory for the dev tools is in /Developer/usr/bin/ — this includes gcc-4.2
  • Do yourself a favor, drop “export ARCHFLAGS=”-arch x86_64”” into your .bash_profile.
• If you’re running homebrew; after the upgrade, I recommend a brew update && brew upgrade
• If you use mercurial — you need to install the updated version found here.
• Just for good measure, do a global (sudo) reinstall of virtualenv, virtualenvwrapper and pip. Make sure they’re pointed at the right default Python (in my case the system one).
• If you are using VMWare Fusion: You really need to be running the latest version of 3.x.
• If you are using Bootcamp, and plan on turning on full disk encryption, see this note from Hacker News (this is why I confine windows to VMware images)

Other than the above; my dev environment pretty much just kept rocking — Lion’s default Python install is a healthy Python 2.7.1 — double nice++ — MacVim, editor of choice just kept plugging away, although I have not tried the “official unreleased” version they have for Lion. I have a slight aversion to running beta builds of my editor.

Some of you are going to run into some annoyances; I can’t help you with all of them, but I can help you with the two interface changes I could not deal with (I actually like all the other ones).

• First; go into system preferences > mission control and uncheck “Automatically rearrange spaces based on most recent use” — trust me, you’ll thank me.
• Second, there’s this … annoying animation when you make new application windows (See the Ars Lion Review | kindle version). The animation offends me on a cellular level. You disable it by running this on the command line:

  defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool NO ; killall Dock

• Show hidden files in finder:

  defaults write com.apple.finder AppleShowAllFiles -bool YES

• Show full paths in finder:

  defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES

• Note: Don’t like mucking around with the command line, or want to have something with a birds eye view that can control just about anything, check out “Secrets” from blacktree. I’m fine with magic PLIST hacks, but this does put a nice UI on all the hacks.
• Also, if you’re a power user, or want tiling behavior for windows — see SizeUp or Divvy (I use Sizeup) both of which are excellent additions to window management, even in Lion. I tend to be anal about window coordination on the screen even with mission control/expose.

Other than that; I’m a happy camper. The new scroll behavior on the trackpads is amazing, if not jarring when using a mouse. Everything just pretty much kept working for me. But then again, I’m pretty easy to please.

See also — “Four Lion Terminal Hacks” from Macworld and “Top 10 Secret Features in OSX Lion” from Lifehacker — if you really, really hate the UI changes, see “How to de-IOSify” from Lifehacker. More cool tips and tricks: “Miscellaneous Lion Tips and Tricks” and also “Miscellaneous Lion Tips and Tricks part two”.

p.s. I originally forgot to mention this — but the full disk encryption in Lion is implemented damn well — it’s easy for users to “get” — seamless, and transparent. In my honest opinion, this is worth the price of upgrading alone if you have a laptop. It’s so well done, and it simply shows how crappy filevault was — it also goes to show that if you make crypto easy and transparent for users, they might actually use it. See the Ars Review for a deeper dive. See also “PGP WDE vs. Lion Disk Encryption”

p.p.s. See this ArsTechnica article about how to create a bootable copy of the Lion install disk you can use to do offline installs for other machines. I’d recommend doing the same for XCode if you have more than one mac to upgrade (or you lack FiOS).

The goal of “speed.python.org” is to port the PyPy Speed Center from “just” a PyPy benchmark tool/suite/site and into one for PyPy, CPython, etc — essentially, a massive source of benchmark information for a range of full Python interpreters. These benchmarks are real world micro and macro benchmarks — something we have Unladen Swallow to thank for — quoting Reid:

I’m also proud of our macro-benchmark suite of real world Python applications which lives on and PyPy uses it for speed.pypy.org. In all the performance work I’ve done before and after Unladen, I have to say that our macro benchmark suite was the most useful. Every performance change was easy to check with a before and after text snippet.

I’ll also note, some of the benchmark suite lives in the Python HG repo — PyPy has done a lot of customizations and tweaks — most of all, they’ve turned it into an awesome site and resource, and having this data, comparing say — Python 2.6/2.7/PyPy/Python3/Jython/etc with historical / per build data (so we could find performance regressions for all of these interpreters will be amazing.

There’s already work occurring on the PyPy side on codespeed (the tool) to make it more generic and usable outside of just PyPy. There’s also two Google Summer Of Code projects which should help — “Turn Codespeed into a multi-project, statistically savvy application” and “Developing a benchmark suite (for Python 3.x)”.

My part to play in all of this was to act, as the magical PSF fairy and sprinkle resource dust into the idea. Fundamentally, I view Python.org not as “CPython.org” — I view it as Python-the-Community.org. I view it as a resource where all interpreters share equal footing, where all of our resources should live and grow (I have a side project for that too…). This single, shared machine is step one in realizing that vision (although I’d be remiss in not noting that plenty of non-cpython things also share our HG repository, and use the mailman infrastructure we have today).

That all said — I finally get to “finish” playing the magical PSF fairy for this, and announce, that due to an amazing donation to the PSF from the HP Open Source Program Office (Thanks Bob!) of a huge machine, and the generous, free racking, stacking hosting and monitoring from the kick-ass team at OSU/OSL we now have this mega machine.

Originally, the PSF was going to fund the purchase of this machine from it’s own pockets — but Van Lindberg connected me to Bob at the HP Open Source Program Office, and bam — magic happened. HP’s generosity got us the following machine:

• DL380 HP DL380G7 X5670 LFF  (2U)
• Dual HP NC382i Dual Port Multifunction Gigabit Server Adapters
• HP Smart Array P410i/1GB FBWC Controller
• 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Registered CAS-9 Memory Kit
• 2x HP 750W Common Slot Gold Hot Plug Power Supply Kit
• HP iLO Advanced including 1yr 24x7 Technical Support and Updates Electronic  License
• 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enterprise 3yr Warranty Hard Drive
• 2   HP DL380 G7 Intel® Xeon® X5680 (3.33GHz/6-core/130W/12MB) FIO Processor Kit

Yes. HP donated a box with 12 cores (24 with hyperthreading enabled) and 16GB of ram, and a high speed RAID to us for this project. And man, does it fly:

Hardware:Processor: 2 x Intel Xeon X5680 @ 3.33GHz (24 Cores), Motherboard: HP ProLiant DL380 G7, Chipset: Intel 5520 I/O + ICH10, Memory: 4 x 4096 MB, Disk: 1200GB LOGICAL VOLUME, Graphics: ATI ES1000
Software:OS: Ubuntu 11.04, Kernel: 2.6.38–8-server (x86_64)

OSU/OSL is going to be our host — they assembled the machine, racked, stacked and they’re going to be our hosts. The entire team — Lance and Peter especially, have been patient, understanding and incredibly helpful. They got this machine up for us today, and helped me work through some auth issues — returning emails within minutes. I’m really looking forward to continuing to work with them — the open source lab at oregon state is an astounding resource for open source. Astounding.

Over the next few weeks, I will be identifying the people who will be taking this from my largely incapable hands and turning it into the resource it will become — in addition to running benchmarks, it’s probably going to help Brett Cannon generate in depth code coverage reports for the standard library — something we intend on breaking out and sharing amongst the interpreters as a common code base.

This is great news — and I’m looking forward to the next few months as the hard part of the project takes off — getting the speed site up and running and working for the community. For the record, I had to test the speed of this beast somehow — so I picked the most resources intensive thing I could think of — translating PyPy using CPython 2.7:

[Timer] Timings:
[Timer] annotate                       ---  732.2 s
[Timer] rtype_lltype                   ---  482.5 s
[Timer] pyjitpl_lltype                 ---  533.4 s
[Timer] backendopt_lltype              ---  146.4 s
[Timer] stackcheckinsertion_lltype     ---   38.9 s
[Timer] database_c                     ---  184.9 s
[Timer] source_c                       ---  344.7 s
[Timer] compile_c                      ---  137.5 s
[Timer] ===========================================
[Timer] Total:                         --- 2600.4 s

As a point of comparison — checking the latest build bot result from PyPy:

[Timer] Timings:
[Timer] annotate                       --- 1452.8 s
[Timer] rtype_lltype                   ---  748.6 s
[Timer] pyjitpl_lltype                 --- 1206.6 s
[Timer] backendopt_lltype              ---  560.5 s
[Timer] stackcheckinsertion_lltype     ---  109.9 s
[Timer] database_c                     ---  712.8 s
[Timer] source_c                       --- 1295.1 s
[Timer] compile_c                      ---  472.6 s
[Timer] ===========================================
[Timer] Total:                         --- 6558.8 s

So, yeah — not apples to apples, but there’s a slight speedup.

We now have the foundation for an amazing, real world bench mark and testing site for Python/Python’s interpreters. This is going to be awesome, and it could not have been done without the work and discussion by the people at the PyCon language summit, the PyPy folks for putting together an awesome site, and without the support of the PSF — but most of all, the generous support from HP and OSU/OSL. My thanks goes out to everyone involved, and I’m honored to be able to help get this project off the ground finally. Let’s show everyone what Python — as a community, working on concert with one another is capable of doing.

PS: Here’s a run of the basic tests from the cpython repo benchmark tool

(photo courtesy of Sean MacEntee via flickr)

Some interesting bits and pieces (leftovers) from the weekend — I have a tendency to queue up a pile of “read later” stuff or emailing myself a pile of things to talk about/link to/etc. Sometimes, I actually get to go through it all. Today — I get to share it to!

First up — Michael Foord (fuzzyman) did two excellent posts — the first is on privacy in Python — not big-P privacy, but rather the programming/object privacy. It’s a good read because it reenforces the point that even if you think you’re being clever and hiding something in a closure to make it private, you can still get access to it. Remember too — dunders (__foo) are just name-mangling. In Python, nothing is private (insert bad tasting joke about Python being facebook here).

Michael’s second post is on the NamedTuple kerfluffle that was stirred up Kristjan Valur’s post on the use of exec() and namedtuple (short version: namedtuple creates a string defining the class and then calls exec to create the object. I think Michael is spot on — I think exec() gets a bad rap frankly, sure — it’s something of a hand-grenade, if you use it wrong, you’re going to get hurt, but in this case I have to agree with Raymond in his comment on bug 3974 — the current implementation is clear and maintainable. I don’t like the sternness of the reply — but he is right. Kristjan’s use-case is an interesting one, but I don’t think it’s one that warrants the removal of the existing implementation. I’m wondering if a “fallback if exec doesn’t exist” is worth inclusion.

Then again, I’ve spoken to people who refuse to use namedtuples because they now know how the sausage is made. I still think the sausage is delicious.

Next is a pretty interesting discussion on a presentation that came out of Pixar on getting over embarrassment in order to get things done — I don’t have much to add above the comments on hacker news, except to say I think the same mental blocks they talk about for animators apply to programmers, writers, etc. We hide from code reviews, we hide our writing until we think it’s “Pitch Perfect” — when in reality, we shouldn’t. We should be sharing, discussing and collaborating earlier, more frequently and more often.

Share and Ship early and often — see also “Don’t Go Dark”.

Finally, I was happy to see the “Quick and Easy Debugging in Python” post from Jeet Sukumaran — it’s always nice to evolve past sprinkling print fairy dust all over your code for debugging — even if we all still do it despite knowing or using PDB. Just to add to his post, if you want to increase your PDB-Fu, I recommend this “Debugging in Python” post by Steve Ferg, and the official documentation for PDB. We should really add a HOWTO for this.

p.s. For additional good-reading, check out “Priorities Don’t Exist in a Vacuum” and “First Care”- while not germane to what’s I’ve already written about, they’re a good essays on priorities. Thanks to the latest “Back to Work” podcast.