Help needed: multiprocessing

by jesse in ,

Originally, this post was going to be much more different than what is has become - the original title was "Failing in Public" - but I don't think "failing" is fair to me personally, or to anyone who has ever helped me, or contributed a patch or a fix to the multiprocessing module.

Yesterday, I made a statement on twitter:

I am officially looking for someone to take over multiprocessing maintenance from me.

Ignoring any comments in that bug; I maintain that a later tweet is still true:

Sometimes good points and poignant criticism can be buried in a pile of crap.

In hindsight; I could have worded the original message differently "taking over maintenance" means that I am, and always have been the sole contributor to the multiprocessing code base, which is patently false. Antoine, and many other python core developers, and people within the community have submitted bug reports, patches, tests and documentation. My words were intentionally harsh - but the direction of that harshness was to me; I feel that as the "leader" (for some measurement of "lead") I have been remiss in my responsibilities and leadership.

Sure; I could be less harsh on myself - but the level of expectational debt that I've incurred against myself for the module and the maintenance has grown, and grown. Even if I find myself leading PyCon, busy as a PSF Director, pushing the core-mentors program, the sprints program, and a lot of other community projects, I am still responsible for the care and feeding for the creature I helped create and birth. I've committed the sin of "going dark".

For some history, see:

Months ago - I spun up the multiprocessing-sig mailing list, in hopes to engage more people - highly active users, interested people, etc to help me pay down the debt. Of course, in retrospect; it's unfair for me to expect anyone but me to help me pay down the debt I've incurred. On the other hand, the responsible thing for me to do - the mature thing for me to do - is to ask for help - not to "wash my hands" of anything, but rather to take this as an opportunity to look as multiprocessing as something greater than what I originally envisioned and submitted to core.

I hear from people every day who are using the module - every day, something I helped birth helps people get things done. Multiprocessing has grown up by virtue of becoming part of Python core, and daily - despite the bugs, the debt and the quirks - it helps developers achieve something they might have otherwise been unable (or at least, had a more difficult time) to do.

The module is expansive - it has pools, tools for distributed programming via managers, pipes for interprocess communications, it's feature set is both large, and ultimately complex in its underpinnings. That complexity - that feature set - is the reason why that debt, the bugs, the quirks has grown over time. If it wasn't being used - I wouldn't have so many emails about it - or bugs filed against it.

So where are we/it today?

Today, multiprocessing has widespread usage - in Python 3, there's actually a new module named concurrent.futures that builds on the building blocks of multiprocessing and threading. Packages like Celery use it extensively (and work around internal quirks). For Python 3 - the sky is the future for what multiprocessing could be - additional functionality, moving parts of it (such as the pool abstractions) into the concurrent namespace, extending and improving the Manager classes, etc. For Python 2.7 - bug fixes, doc fixes only.

If you search the Python bug tracker for the word "multiprocessing" regardless of assignee, you'll get 119 hits. That's right; 119 - not all of them are multiprocessing bugs - and many of them are dupes, or fixed in recent versions. What that query gets you is an idea of the debt that has to be paid down and resolved. Each one of those bugs needs to be looked at, reproduced, de-duped and patched. Some of them may be documentation issues, some are pretty hairy (like the aforementioned as well as and

What I asking for - rather than washing my hands of anything, or any attempt to absolve myself of responsibility, is for help. I am stretched thin - too thin to do this myself, or to be the only person who can maintain, understand or work on this module. It's too big for that, it's too important for me to be the arbiter of it any longer. It's bigger than me.

Aside from the bug queue; there's a short list of things that need to be done - the docs need to have a fresh, hard set of eyes on them, there are things (behaviors, features) that are undocumented. The test suite needs a complete overhaul - when I inherited the code, this is the first thing I should have done - but I didn't. The problem is that the test suite is mired in magic and complexity, and without an expansive, maintainable test suite, I don't feel confident that the bug list can be addressed with confidence.

So, I come to you with my hat in my hands, a humbled man. It's unreasonable for me to ask for others to "pay down" the debt I've incurred; but it's irresponsible, immature and misguided of me to think that I alone, or any single person can go at this alone. So I need your help - and, if in time, someone choses to be the "leader" for the module, then I will gladly step back. Until then, I will try to continue to be a guiding hand and at least point people in the proper direction, commit patches, etc.

If you are interested; please speak up - or just wade into the bug queue. You can sign up to the multiprocessing-sig list, and ask questions there, or if you're new to core Python, and want some additional mentorship, check out the Python Core Mentorship program - that list serves as a gentle and polite, welcoming introduction to core development. No question - no matter how green - is off limits, and it's already got an excellent track record of helping people get up to speed.

In closing; I'm going to apologize - we all know lives change, careers change, and interests change. All of these things have happened to me, but in changing so quickly and taking on different roles, I left something important behind. In doing so, I have done a great disservice to you, the community and users.

I will also thank you; without you - the users, current and future helpers, multiprocessing wouldn't exist or be relevant in any context. Without you, I wouldn't have the drive to even write this post, fight to get multiprocessing into the std lib to begin with, or perform any of the other roles I do.

So; thank you.

Quick Python/Developer tips for OSX Lion

by jesse in ,

Lion king 5067

Yes; of course I upgraded to OSX Lion on day one. To quote myself from twitter:

I am happy in the warm coziness and stark whiteness of Steve Jobs' monoculture.

Regardless of that though, I had very few hiccups with Lion itself - but a few things you need to deal with going in:

  • Be on a high bandwidth connection. The downloads you have to make are huge.
  • XCode 4.x is now free in the App Store - you need this - the first thing you need to install after Lion is the latest version of XCode from the App Store, if you do this, all your virtualenvs, your homebrew environment (at least mine) just Keep Working. Save yourself some pain.
    • Side note: @kennethreitz, gentleman scholar, has actually done a custom osx-gcc-installer - this contains a system install of GCC and all the goodness you need (such as install_name_tool) for Python hacking. So you might be able to skip the massive XCode install.
    • When installing XCode, for some unknown unholy reason, if you have not quit itunes, and itunes helper (see activity monitor) prior to starting the XCode installer, the install will hang. Do yourself a favor and kill it with fire.
    • Remember; the binary directory for the dev tools is in /Developer/usr/bin/ - this includes gcc-4.2
    • Do yourself a favor, drop "export ARCHFLAGS="-arch x86_64"" into your .bash_profile.
  • If you're running homebrew; after the upgrade, I recommend a brew update && brew upgrade
  • If you use mercurial - you need to install the updated version found here.
  • Just for good measure, do a global (sudo) reinstall of virtualenv, virtualenvwrapper and pip. Make sure they're pointed at the right default Python (in my case the system one).
  • If you are using VMWare Fusion: You really need to be running the latest version of 3.x.
  • If you are using Bootcamp, and plan on turning on full disk encryption, see this note from Hacker News (this is why I confine windows to VMware images)

Other than the above; my dev environment pretty much just kept rocking - Lion's default Python install is a healthy Python 2.7.1 - double nice++ - MacVim, editor of choice just kept plugging away, although I have not tried the "official unreleased" version they have for Lion. I have a slight aversion to running beta builds of my editor.

Some of you are going to run into some annoyances; I can't help you with all of them, but I can help you with the two interface changes I could not deal with (I actually like all the other ones).

  • First; go into system preferences > mission control and uncheck "Automatically rearrange spaces based on most recent use" - trust me, you'll thank me.
  • Second, there's this ... annoying animation when you make new application windows (See the Ars Lion Review | kindle version). The animation offends me on a cellular level. You disable it by running this on the command line:
    defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool NO ; killall Dock
  • Show hidden files in finder:
    defaults write AppleShowAllFiles -bool YES
  • Show full paths in finder:
    defaults write _FXShowPosixPathInTitle -bool YES
  • Note: Don't like mucking around with the command line, or want to have something with a birds eye view that can control just about anything, check out "Secrets" from blacktree. I'm fine with magic PLIST hacks, but this does put a nice UI on all the hacks.
  • Also, if you're a power user, or want tiling behavior for windows - see SizeUp or Divvy (I use Sizeup) both of which are excellent additions to window management, even in Lion. I tend to be anal about window coordination on the screen even with mission control/expose.

Other than that; I'm a happy camper. The new scroll behavior on the trackpads is amazing, if not jarring when using a mouse. Everything just pretty much kept working for me. But then again, I'm pretty easy to please.

See also - "Four Lion Terminal Hacks" from Macworld and "Top 10 Secret Features in OSX Lion" from Lifehacker - if you really, really hate the UI changes, see "How to de-IOSify" from Lifehacker. More cool tips and tricks: "Miscellaneous Lion Tips and Tricks" and also "Miscellaneous Lion Tips and Tricks part two".

p.s. I originally forgot to mention this - but the full disk encryption in Lion is implemented damn well - it's easy for users to "get" - seamless, and transparent. In my honest opinion, this is worth the price of upgrading alone if you have a laptop. It's so well done, and it simply shows how crappy filevault was - it also goes to show that if you make crypto easy and transparent for users, they might actually use it. See the Ars Review for a deeper dive. See also "PGP WDE vs. Lion Disk Encryption"

p.p.s. See this ArsTechnica article about how to create a bootable copy of the Lion install disk you can use to do offline installs for other machines. I'd recommend doing the same for XCode if you have more than one mac to upgrade (or you lack FiOS).


Thank you - The impossibility of "It's going to be ok".

by jesse in

Let me start off by saying that while this post is largely not Python related although it is slightly related in the fact that I talk about the Python community later on. Largely this post is about my family and some of the troubles that we've been going through, and how it has affected me.

If you're looking for a technical Python article then you should probably move on. I realize that it has been awhile since I've been able to do a deep post on a pure Python topic; for that I apologize. This post talks a lot about community towards the end; so it may remain of interest.

I've been writing, and re-writing this post in my head over and over again - some of it due to the fact that the problems we've been dealing with are just not something I've ever dealt with before, but also I didn't quite know how to put things into words. Throw in a healthy dose of "1 month old induced sleep deprivation" and you have a combination for scrambled brains and scattered thoughts.

Mostly, it is the emotional aspect - more importantly I've been sitting here rewriting this post over and over and over again because it is never easy for parents and it is especially not easy for me, just because of who I am, to sit down and put to words the experiences of the past month. My wife, Dusty has has organized and posted her views on what's happened over on her blog - I encourage you to go read those:

She's done an excellent job putting her thoughts down eloquently - far better than I've managed. I don't think I've stopped "working" in any sense of the word - you get busy just living, you get busier with one child - with a second child (especially an infant) - no one gets free time. Throw in a busy job at a startup, chairing PyCon, and a handful of other things and I'm pretty happy to be sitting here correcting a horribly dictated blog post.

So, backing up - on June 2nd, we welcomed Addison Joy to the world - she's the second of our beautiful children. As many of you might know - or - you've read my wife's posts - the pregancy was pretty hard on everyone involved, but especially my wife. It was touch and go and that is stressful enough.

But, so, AJ was born - and at first, everything seemed to be fine. 10 fingers, 10 toes and pooping - that's sort of what you hope for in a newborn. We took her home, she saw her pediatrician, and that was that.

Well, no. Around the time Addison was three weeks old (shortly before my first child's birthday) my wife Dusty started noticing that Addison was behaving erratically/oddly - and if you have any experience with infants, you'd know how hard it is to actually determine "odd" behavior. Almost everything they do is odd, down to timing exactly the worst moment when to spit up on you (point of fact - it is after you've showered, and are walking out the door).

In this case, the odd behavior my wife noticed was actually a pattern - and that's when you need to worry. You want consistency in certain areas, you want to see continual improvement, you want them to consistently eat, poop and sleep. However, a pattern of odd movements tipped my wife (who is a fantastic analyst) off that something was not quite right.

What my wife found was that Addison for periods of time anywhere from 1-2 minutes her eyes would slit and roll back and she would freeze up. The best way to describe it is it was almost as if she would just "check out" - as if someone hit a power switch.

Her eyes rolled back and her body would go stiff - her breathing would go robotic. My wife didn't want to scare anyone come off as the the crazy overprotective parent but it was happening more more frequently and it was happening in clusters. Basically, Addison would have these spells in groups of 3 to 4 and she was having them more frequently than you'd care to imagine through the day and night.

As this pattern began to emerge and my loving wife, the analyst, started to see the trends - she started to dig around search and ended up finding a video on YouTube of exactly what we were seeing. Unfortunately, the video matched what was happening - what we were both now seeing that the pattern had become clear - and the video was of a small infant having seizures. Patterns, especially hidden ones (like the arrow in the Fedex logo) are very hard to un-see once you've seen them, and this was no different. We knew something was up.

She passed that video, and all of our suspicions to our primary pediatrician - someone from the old-school of pediatrics. Within an hour, he had already set us up with a rush appointment at one of the best pediatric hospitals in the United States - Tufts Floating Hospital for Children. Luckily, it is about an hour away from where we live.

Within a few days, we were sitting and talking to some of the best child/infant neurologists in the area. No one was fooling around - the turnaround time to us identifying something and us sitting in a room with at least two top notch neurologists was amazing.

The staff has been amazing: from our first visit they have been kind, courteous, they have helped us manage our four year old Abby who had to come along for the ride. They answered all of our questions, encouraged us to get second opinions, etc. The first meeting we had with them, they looked us flat in the eyes and said "you're not leaving here without a plan". God help me, I could have hugged them at that point.

They watched some of the videos of the episodes my wife had captured on her iPhone - we'd been instructed to record as much as possible. We discussed the episodes and their "presentation" (what they look like, how Addison moves during them, which way do the eyes roll back, etc) and so on.

Then, in a flash, it was off to the EEG - this is where you, as a parent, feel largely like a useless appendage. They take your 4 week old daughter and stick little electrodes to her head and watch the electrical impulses in her brain fire. Shortly after the EEG my wife indicated to me that she knew they had seen something but they were being relatively tightlipped - they immediately sent us off to another department to get a sonogram of Addison head. A sonogram is essentially an ultrasound - they were looking for physical abnormalities, calcium deposits, water on the brain and tumors. Luckily, the sonogram came up clean.

Shortly after the sonogram the team of doctors that we been working with came in and told us flat-out that the EEG had been abnormal. What this means is that they noticed distinct abnormal electrical behavior in both sides of Addison's frontal lobe. Based on this, they would immediately assume she was having seizure activity and that epilepsy was a very real possibility. Therefore in order to get the situation under control we would immediately put Addison on to a drug to help control seizures - starting with a small dose, and working our way up until the seizures stopped.

Now when dealing with epilepsy in infants there are only two real drugs that doctors are willing to recommend the first is phenobarbital, which has some pretty awful side effects (and a tranquilizing affect on the infant), such as causing some serious liver problems, etc. The good news is that phenobarbital has about 150 years of use behind it - doctors know it pretty well, and it has known to help with epilepsy quite well. The problem is the side effects - well, Dusty and I agreed the side effects were too much.

The second drug is Keppra - this one is not as old as phenobarbital - but is also known to work on infants with seizures. The side effects on this one are less, but not entirely non-existent - one of the nastier ones is crankiness from the infant. Addison is a pretty chilled out baby and so giving her something that would make her angry all the time wasn't something we were looking forward to, but we went down that road.

So the plan was this: in about two weeks, bring Addison back for a 24 hour EEG so they could capture more data and record her on video for that time, so they would have as much data as they could compile. The following week would be an MRI so they could look deeply into things to verify everything was structurally sound

From the time we left the hospital, throughout the weekend, etc - the staff and doctors were emailing us, calling us and checking in on us. They were attentive, kind and making sure everything was going ok with Addison as we steadily increased her dosage of the drug - .2 mL, .4 mL - to eventually .6 mL.

As the hours passed - you could actually see the episodes changing - they were getting longer, but less "twitchy" - the clusters were shorter/different. It crazy - you could actually see her body's reaction in real-time. Unfortunately, we could also experience her attitude changing in real-time as well.

So here we are, two parents - two kids. I've got a full time job (and full time volunteer work on the side) and my wife had just been settling down to being the best damned stay at home mom you've ever seen. We're running on little to no sleep and bam. A bus slams into us. Epilepsy. Our beautiful newborn baby girl probably has a form of epilepsy.

What did we do? Did we cause this? Your fears, doubts and every little thing comes rushing into this mental void left by the explosion of this new reality.

And so there we are - drugging a newborn in hopes to kind of get these things under control and hoping for the best. As a dad, the typical thing you want to do - your typical guy response - is to fix the problem. You have to have an answer - any answer.

You want to fix your helpless daughter sitting there looking off into space.

That's not how it works though. You have to sit there and watch, and wait. You have to keep your poker face on when the doctors tell you your kid could have an incurable disease - or something she could grow out of in a few months - no one knows, and it is impossible to tell in children this young.

It is not the problem that kills you inside - it is the uncertainty - it is the not knowing and the feeling helpless to do anything. Throughout all of this, my wife - Dusty - has been my hero. She's pushed through, asked all the right questions, pushed the doctors - and me - to do better, answer more, to step up and beyond.

The following week, the MRI was shown to be clean - another exhale of breath - that means no surgery (but also, no "easy answer") - go home, keep her on the Keppra, the 24 hour EEG is coming up. Watch and wait.

Then, the Wednesday I finally decide to scrape my body out of bed at 5:30 in the morning (this is really hard with newborn) and go to Yoga, things go sideways. While I'm in yoga - my cellphone is shut off (it doesn't work well in the heat), and it is quiet. Little did I know, that while I was there, my wife was in the process of dialing 9-1-1 - Addison had fallen into a grand mal seizure, and she could not pull Addison from it/snap her out of it. The seizure went on for over 8 minutes. Warning signs.

By the time I was out of class I had a mailbox of messages and a lot of guilt - an hour had passed since my two daughters and wife had been taken via ambulance to a local hospital in their pajamas. The plan was to transfer Addison to Tufts in Boston ASAP. I had some catching up to do, and some apologies to make. I still don't feel right thinking about it - I let my family down, while I was off working out.

In the ambulance, the paramedics were able to snap Addison out of it, which was good news. Fast forward through Addison being transferred to tufts all by herself (family cannot travel in the transports if they have children - and I had yet to show up to the hospital) and me driving pretty illegally to come and get my wife and Abby and shoot downtown to see if we could beat Addison getting here.

Before this rush to the hospital visit - the diagnosis had been "general non-specific seizures" - doctor-speak for "she's too small for us to pinpoint it, but something ain't right". Then, here we are - rushed in, a video camera pointed at my 5 week old daughter's head, electrodes cemeted onto her head, rolling off the bed in a bundle to the EEG machine, an IV plugged into her foot.

Before we could say boo; they immeditely doubled the dose of the Keppra she had been on (from .6 mL to 1.2 mL) in hopes to bring the episodes under control. That started the clock - the first day there, my wife and I and Abby were there most of the day. I sent them home to get food, and a change of clothes for me - I'd be pulling the overnight shift - and so it went for the next 48 hours.

Sleeping two nights in a hospital cot with your newborn hooked up to crazy things next to you is pretty much a weird experience.

And herein lies the rub; they checked the first 24 hours of the EEG pretty quickly, and it came back up clean. They had video of Addison having the episodes, but they were not appearing on the EEG. Everyone pretty much assumed that the Keppra was keeping things under control - so for the second 24 hours - they took her off the Keppra, but kept recording. The problem of course, is that Keppra takes days to flush from your system.

So, another set of clean EEGs - this has all of us flummoxed - everyone there had video and had seen Addison slipping into these episodes. But without the EEG data to back it, there was nothing solid to poke at. It gets more frustrating when whatever small, painful answer you had is clouded over because they can't prove it (or even disprove it).

And so, they sent us home - off the Keppra, with Addison hooked up to all the electrodes cemented to her head, and a portable EEG machine recording the entire time, like a little black box. They removed that last Monday. Then last thursday, she was back in for another EEG that came up clean. The catch? They have video of her having the episodes right on camera while the EEG shows she's not asleep/out to lunch, just "frozen".

And now, here we are - diagnosed with a "seizure disorder" - we're no closer to an answer than we were before. it is hard - with children this young, it is actually really difficult to get reliable tests from them, and some of the other tests they can do - such as spinal taps - could cause more harm to her than good. We're in "wait and test more mode". Babies' brains and nervous system mature/change on a weekly basis  - so what might be true this week, may not be true the next. It makes it very, very hard to diagnose problems unless they're glaringly obvious.

Last week we involved early intervention which is actually a government program that will come and help out your child and help you with coping and identifying issues and making sure you know she's hitting all of her developmental milestones.

They did their initial evaluation - which, again, given Addison's early age means there's a lot of fudge in the numbers however, early intervention found that Addison is 30% behind the average on three of the key milestones (development stages). All of the milestones she is behind on happen to be ones controlled largely by the frontal lobe - where we have seen abnormal activity. Her physical milestones are on target - the others, well - 30% behind.

Now - that could be nothing. Babies develop at different rates all the time, Addison could just be slower developing in those three areas; however this coupled with the episodes she's been having could indicate a problem in her frontal lobe which may not fully present itself until she's older.

This also means a lot of work - we have to be a lot more focused on these milestones/stages than most parents. Talk, play music - things you already do as a parent with infants, but now we have to watch her reactions (with the help of doctors and nurses) like a hawk. We have to make sure she does not halt developing, and does not in fact go backwards in any way.

We are in for regular visits from nurses, regular visits (EEGs) with the Neurologists so that they can make sure she hasn't drastically changed. Right now, they can't put her back on the anti-seizure medication, without proof on the EEG, even with video and hands on proof of the episodes, having her on the medication without the EEG data to solidly back an epilepsy diagnoses could harm her more than help her - even if we know it does help the episodes she is having now.

So here we sit - everyone knowing - from her pediatrician, to the neurologists, to the nurses, that something is wrong. We can't ask Addison what's wrong, what she's feeling, or anything else. All we get to work on are facial expressions and crying. There's nothing you can do as a parent - we have to sit and watch her light switch shut off and her tiny hands shake, there's no drugs, there's nothing but testing, and waiting, and watching.

It brings us to an unpleasant place - a long road of doctor visits, not knowing, waiting and hoping. Hour long trips into the city to the hospital through Boston's lovely traffic. No answers, just negative ones - "it is not x, or y - right now". There's nothing proactive to be done - only reactionary. Watch and wait and react. If things get better - we rejoice, if things get worse, we react. Doctor visit after doctor visit, EEG after EEG.

So the reason I'm writing this post kind of a part catharsis and part update to a lot of friends I, and my family have out there. We're lucky - I have a great job I love, we have great medical insurance and the best hospitals in the country nearby. We have a lot of things going for us.

Despite that, it is still trying. Mentally, physically - time and attention - financially. It is not easy - but many people have it worse.

Now, if you've been following me on twitter, or google+ - much of this comes as no surprise to you. Many of you - well, most of you - are members of the Python community, the Python Software Foundation, you help with PyCon - somehow, I, or my family know you.

And we've been touched by you.

Something I didn't count on, baring my soul on twitter, or google+, or Facebook was the overwhelming and humbling support my family and I would receive from all of you. The well wishes, the emails - cards from all over the world wishing us well and offering us your hopes and prayers.

Some of you may have met me - almost none of you have met my wife and family - and yet the outpouring of support from the Python community has humbled me and brought me to my knees in thanks. It brings tears to my eyes just thinking about the generosity that has been bestowed on us by people inside of this community. Something as small as a card - a box of crayons and a coloring book for my oldest daughter - it has helped my family and I and touched us in a way I don't think we've ever been touched.

It is amazing to me that I can admit to hurting or going through something like this and people all over the world will immediately start sending the resources on where to look for information or who to talk to or specialists that they know, contact information for family members that they know who have experience with epilepsy or seizures.

It has been amazing to me the amount of support that I've gotten from a community that's based on a programming language; we are all engineers and it is not something that you'd necessarily expect.

Except for the fact that at the end of they we are all humans and as God as my witness I have to say that all of you Python hackers, and friends - both internet and in person are the best, most touching human people I've ever had the luxury of dealing with in my entire life.

I've had the honor of standing up on stage at PyCon addressing 1300 Python hackers and that was stunning and humbling in and of itself - but to have people who you mainly talk to on the Internet and primarily through mailing lists and code, twitter and google - who in one moment might be arguing with you violently about something about packaging, test syntax or concurrency - to have those same people turn around and lift your family up in your time of need - it is beyond words. This Python community - our community - is something to be cherished.

There have been several occasions where out of sight of my family and the rest the world I've cried because of what the community has done for us in how much it has supported us. Our friends, this community, our family - I can't describe it.

Something as small as getting a package in the mail that has a couple of stuffed animals for my oldest daughters - that just brightens thier day, a card wishing us well or an email or tweet to my wife - it makes the day brighter. In the past month those bright days have been rough and very hard to make as a father and it has been hard to get up in the morning in the face of "ship this code, ship this conference, figure out what's wrong with your daughter, keep the boat afloat". All of the support we have gotten has helped prop me, and my family, up. It has helped us make what could have been crushing days not so dark.

We don't have family in the area (at least none remotely close) and it has been amazing having people reach out to you from completely different countries when your next door neighbor doesn't even know something is going on.

It is amazing, it is humbling - and I just want to say thank you from the deepest part of my heart. There might be more tough times coming up ahead - I don't know - but it is my job as a father to get up everyday put on that game face and ship that code, fix those bugs, and ship a conference, make money and be a dad - it is my job as a father to do the impossible everyday because that is what I do.

Its my wife's job to get up, put on a game face and do the impossible everyday, she is my hero, she doesn't "get" to go to work, a small vacation I get every day. She is in the fray all day, every day, doing the impossible.

The impossible is already hard enough - raising children, holding down a full time job and juggling a hundred other things - things parents do every day is hard enough but it is our job as parents and our duty to stand up straight and do everything that we can for our children.
In the case of our child - it is our job to do the best damn job that we possibly can, give her the best care, the most love, and to hope and pray that everything will come out all right.

And that's the odd thing throughout all of this deep down inside - if you ask my wife - I'm a starry eyes optimist, for example I believe that everyone no matter who they are or their attitude or background can be a contributor to not just Python - but its community and vibrancy as well. I believe that human beings are intrinsically good creatures - I believe that we as people and as humans are capable of doing the impossible and the incredibly difficult every single day.

Being an optimist I have to look at the situation with my daughter Addison and tell myself that it will be okay. I have to keep telling myself this time and time again because I have to reinforce it in my head that that everything is going to be okay - you know she's still a fantastic baby, she still sleeping large chunks of the night, she's eating, she interacts with us.

She's a great baby and daughter - both my daughters are great and amazing - just every so often she checks out. It is a very real possibility that in three months Addison could grow out of this and all this fright and energy was "for nothing" - it is also a very real possibility that in three months things could get fantastically worse.

It is possible it is just due to the hard pregnancy that my wife had with the hospital stays and everything else and that Addison is just catching up developmentally to things that she should've had when she was born and now chalk it up to "weird baby things"

I have to keep telling myself that she is going to grow out of it, that she is just going to be fine, and you know some part of me when sitting in a dark room might whisper to me that I'm lying to myself that it is not going to be okay and things are only going to get worse but I know that's not the case.

Parents go through things every day that are much, much worse than what my wife and I have gone through. No one wants to go through that we've gone through but other parents go through much worse and they adapt and they raise some of the best children that you've ever seen. Some of my heroes have raised children with problems such as epilepsy or autism and their children are amazing, much more amazing than me. So despite what Addison may or may not have, despite not knowing and just having to sit back and watch as something happens that I have no power over - despite all that, I know things will be okay.

Because I know as a parent and as a human, it is my job to get up every day and do the impossible. It is my job as a parent to raise the best children - not perfect children - the best children that I can and do right by them. It is my job to teach them, to love them and do everything in my power to either heal them or help them cope with whatever may be wrong. It is hard, but it is not impossible - but the impossible is our job.

Reach out to those you know are hurting, or are scared - or heck, people with one month olds that keep them up all night - wish them well, send them a card, or an email. Even if you only know them through twitter, python or something else - maybe they're a kernel hacker, maybe they're someone working on the next big deployment tool - it doesn't matter who they are or what they're doing Engineers are Humans (as Ned so aptly put it) - and all of those people trying to achieve, and fight the impossible are even more than that.

So while some part of me is scared, is frightened, is angry - that's not the part that I can listen to at all - that's the part that's been pushed into the darker corner of my brain not only because of my optimism that's baked into my brain but also thanks to the support of our friends both inside of, and outside of the Python Community and our family.

Because of the support, because of the well wishes, because of the kind words because of everything that's been done for us, on behalf of us, and because we can do the impossible.

Because of all this I know everything's going to be okay.

Thank you. From my family, and the bottom of my heart.


p.s. There are many families out there - probably people you know - who have children with issues, or who are struggling with issues such as these who are much, much worse off than my wife and I. My family, while going through this, is very lucky. We've been blessed with wonderful friends, this community and our family. I encourage you again to look around and see those people who are much more deserving than perhaps I, or my family is, and to help them. Even if it's just a card; everything helps.



Announcing the new machine!

by jesse in , ,

I hinted at this on twitter earlier - but now that I've had a chance to ssh into the box finally, and, well - run some benchmarks and translate PyPy - I am very happy to announce that the PSF has acquired quite the beefy machine for the planned site discussed at the Python VM summit at PyCon 2011.

The goal of "" is to port the PyPy Speed Center from "just" a PyPy benchmark tool/suite/site and into one for PyPy, CPython, etc - essentially, a massive source of benchmark information for a range of full Python interpreters. These benchmarks are real world micro and macro benchmarks - something we have Unladen Swallow to thank for - quoting Reid:

I'm also proud of our macro-benchmark suite of real world Python applications which lives on and PyPy uses it for In all the performance work I've done before and after Unladen, I have to say that our macro benchmark suite was the most useful. Every performance change was easy to check with a before and after text snippet.

I'll also note, some of the benchmark suite lives in the Python HG repo - PyPy has done a lot of customizations and tweaks - most of all, they've turned it into an awesome site and resource, and having this data, comparing say - Python 2.6/2.7/PyPy/Python3/Jython/etc with historical / per build data (so we could find performance regressions for all of these interpreters will be amazing.

There's already work occurring on the PyPy side on codespeed (the tool) to make it more generic and usable outside of just PyPy. There's also two Google Summer Of Code projects which should help - "Turn Codespeed into a multi-project, statistically savvy application" and "Developing a benchmark suite (for Python 3.x)".

My part to play in all of this was to act, as the magical PSF fairy and sprinkle resource dust into the idea. Fundamentally, I view not as "" - I view it as I view it as a resource where all interpreters share equal footing, where all of our resources should live and grow (I have a side project for that too...). This single, shared machine is step one in realizing that vision (although I'd be remiss in not noting that plenty of non-cpython things also share our HG repository, and use the mailman infrastructure we have today).

That all said - I finally get to "finish" playing the magical PSF fairy for this, and announce, that due to an amazing donation to the PSF from the HP Open Source Program Office (Thanks Bob!) of a huge machine, and the generous, free racking, stacking hosting and monitoring from the kick-ass team at OSU/OSL we now have this mega machine.

Originally, the PSF was going to fund the purchase of this machine from it's own pockets - but Van Lindberg connected me to Bob at the HP Open Source Program Office, and bam - magic happened. HP's generosity got us the following machine:

  • DL380 HP DL380G7 X5670 LFF  (2U)
  • Dual HP NC382i Dual Port Multifunction Gigabit Server Adapters
  • HP Smart Array P410i/1GB FBWC Controller
  • 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Registered CAS-9 Memory Kit
  • 2x HP 750W Common Slot Gold Hot Plug Power Supply Kit
  • HP iLO Advanced including 1yr 24x7 Technical Support and Updates Electronic  License
  • 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enterprise 3yr Warranty Hard Drive
  • 2   HP DL380 G7 Intel® Xeon® X5680 (3.33GHz/6-core/130W/12MB) FIO Processor Kit

Yes. HP donated a box with 12 cores (24 with hyperthreading enabled) and 16GB of ram, and a high speed RAID to us for this project. And man, does it fly:

Hardware:Processor: 2 x Intel Xeon X5680 @ 3.33GHz (24 Cores), Motherboard: HP ProLiant DL380 G7, Chipset: Intel 5520 I/O + ICH10, Memory: 4 x 4096 MB, Disk: 1200GB LOGICAL VOLUME, Graphics: ATI ES1000
Software:OS: Ubuntu 11.04, Kernel: 2.6.38-8-server (x86_64)

OSU/OSL is going to be our host - they assembled the machine, racked, stacked and they're going to be our hosts. The entire team - Lance and Peter especially, have been patient, understanding and incredibly helpful. They got this machine up for us today, and helped me work through some auth issues - returning emails within minutes. I'm really looking forward to continuing to work with them - the open source lab at oregon state is an astounding resource for open source. Astounding.

Over the next few weeks, I will be identifying the people who will be taking this from my largely incapable hands and turning it into the resource it will become - in addition to running benchmarks, it's probably going to help Brett Cannon generate in depth code coverage reports for the standard library - something we intend on breaking out and sharing amongst the interpreters as a common code base.

This is great news - and I'm looking forward to the next few months as the hard part of the project takes off - getting the speed site up and running and working for the community. For the record, I had to test the speed of this beast somehow - so I picked the most resources intensive thing I could think of - translating PyPy using CPython 2.7:

[Timer] Timings:
[Timer] annotate                       ---  732.2 s
[Timer] rtype_lltype                   ---  482.5 s
[Timer] pyjitpl_lltype                 ---  533.4 s
[Timer] backendopt_lltype              ---  146.4 s
[Timer] stackcheckinsertion_lltype     ---   38.9 s
[Timer] database_c                     ---  184.9 s
[Timer] source_c                       ---  344.7 s
[Timer] compile_c                      ---  137.5 s
[Timer] ===========================================
[Timer] Total:                         --- 2600.4 s

As a point of comparison - checking the latest build bot result from PyPy:

[Timer] Timings:
[Timer] annotate                       --- 1452.8 s
[Timer] rtype_lltype                   ---  748.6 s
[Timer] pyjitpl_lltype                 --- 1206.6 s
[Timer] backendopt_lltype              ---  560.5 s
[Timer] stackcheckinsertion_lltype     ---  109.9 s
[Timer] database_c                     ---  712.8 s
[Timer] source_c                       --- 1295.1 s
[Timer] compile_c                      ---  472.6 s
[Timer] ===========================================
[Timer] Total:                         --- 6558.8 s

So, yeah - not apples to apples, but there's a slight speedup.

We now have the foundation for an amazing, real world bench mark and testing site for Python/Python's interpreters. This is going to be awesome, and it could not have been done without the work and discussion by the people at the PyCon language summit, the PyPy folks for putting together an awesome site, and without the support of the PSF - but most of all, the generous support from HP and OSU/OSL. My thanks goes out to everyone involved, and I'm honored to be able to help get this project off the ground finally. Let's show everyone what Python - as a community, working on concert with one another is capable of doing.

PS: Here's a run of the basic tests from the cpython repo benchmark tool

HPR D B RGB 150 SMOsl logo