jessenoller.com

Breaking up is hard to do (because I got a new job)

Earlier this week - I made the rather difficult decision and resigned from my current job. Yes - I have a new one in the bag.

The job I resigned from has been one of the best, most rewarding jobs in my career. I've had the luxury of working with some of the best engineers I have ever met, on a problem/solution that is, by far, the best in it's class.

I've been working with what I count as my second family for close to five years. You don't walk away from something like that lightly, or without good reason.

My reason was simple: I enjoy startups. I love blazing trails, exploring technology. I love pushing boundaries and making something tangible and new where there was nothing before. It's why I love software.

Some people would say that software is not a painting - and to an extent, they are right. But the pride you feel when you see your software come alive, get adopted and grow day after day is much the same feeling of accomplishment and pride you get when you do finish a work of art.

Personally, I feel that startups engender the wild west - all ideas are open and ready to be explored, much of the territory is uncharted, and you remain (in many cases) unencumbered by processes, bureaucracy and history.

That all being said - it's not without an aching heart that I say good bye to my friends, colleagues and the product I helped usher into the world. Both the product, and the people remain best-in-class, and I wish both the company, and the people the best in the future.

The great (and sometimes bad) thing about startups though, is the fact that if you live in a place where startups are always coming and going (i.e: Boston) then sooner or later, you're going to end up working with many of the same people again. Due to this, I'm ultimately OK with saying good bye to many of the coworkers I have today, because I will see them again in due time.

That all being said - I have a new job (obviously). Later this month I will begin working for an Acton, MA startup called Blackwave. I've had the pleasure of meeting the people there and exploring the technology, and I have to say I got that old spark back in my head. The spark that keeps you up at night thinking about how to solve The Problem and how to test The Problem. The one that makes you get up early to write notes down to yourself like "if we simply change x here and here, then in theory we can cut our testing to y..." and so on. It's the spark of inspiration.

In speaking with them - I wanted to solidify something that's been chewing at me for a little bit - and that's having some amount of freedom to contribute to open source - in my case, Python-Core and python-related projects. I'm happy to say, Blackwave was more than willing to express their support and openness to contributing to the community. I can't go into details - but with any luck, I should be able to be a lot more open with much of my work, and also give a lot back.

Again - with that all being said - I will take this one opportunity to say - yes, Blackwave is hiring. If you're a python/java/c/c++ person, or you just love doing QA and testing and are looking for a small, fast moving team - and live in the Boston/Metrowest area (southern New Hampshire is OK) - go ahead and drop me an email. I'm definitely putting feelers out for any Pythonistas in the area who are looking around. I'm doubly interested in Pythonistas with a QA background to help focus on test engineering/automation.

Send your resumes to me (jnoller at gmail dot com)!

I'm looking forward to the upcoming months with excitement.

Hi, my name is Jesse and I abuse list comprehensions.

I just found myself being quite abusive.

I have a series of dict{key:[list]) style objects, and I wanted to unpack it, convert it to a string, etc. (yes, variable names have been changed to protect the innocent)

 
return ',',join(['{{whizbang|%s}}' % i['foo'] for i in [ i for i in myDict[keyname]]])
 

Now, this could only be better if I:

• Included a lambda
• Had another nested comprehension

Other than the fact that what I wrote above is next to untestable - it's also hard to understand unless well, you're me. In order to debug it, you'd have to pprint the inbound dict and slowly unravel (from left to right) the operations being done.

Luckily, there's no math there!

So, I "cleaned" it up:

 
whizlist = []
for item in [k for k in myDict[keyname]]:
    whizlist.append('{{whizbang|%s}}' % item['foo'])
return ','.join(whizlist)
 

Hmm, I still seem to be juggling chainsaws here. Let's go for the "easy" button:

 
whizlist = []
for item in myDict[keyname]:
	whizlist.append('{{whizbang |%s}}' % item['foo'])
return ','.join(whizlist)
 

I could argue that the first is "faster" (although I'd need to prove it) - but ultimately the final "un-optimized" version is more pythonic.

Miscellanea and a Pycon closeout

Since being back in "the real world" - things have been crazy. I'm working on a PEP (albeit slowly) and trying to ramp up on python core-development while simultaneously trying to work on the safetheading stuff and several PyMag articles.

Did I mention that I can't work on any of this during the day? *head explodes*

Oh, and there was Easter Dinner to attend to as well? I do so love cooking.

I envy the Googler's out there and the others who get an allotment of time to "contribute to open source" or explore other technical avenues for many reasons, the obvious of which is "having more time to do something else".

To boot, I've been compulsively saving things to read/catch up on in my newreader. I'm up to about 205 items to cycle through.

All the whining done - I never did get to "close out" PyCon.

The last two years I've gone, I never stayed long enough on Sunday to see or do anything useful. This year, I wanted to stay through the core sprint, but alas - I had to go back Monday.

I was at least able to finish out the con, see some great talks and do a lightning talk on concurrency stuff. I also got to sit through Brett's intro to core development (which was great by the way) and try to get up and hacking on core.

I unfortunately spaced out during most of the morning keynotes (except for the OLPC update) as the mozilla-world isn't my particular cup of Joe. In addition to that, I got tapped by Jacob Kaplan-Moss to do a lightning talk (they had an open slot) - I said yes, and promptly went heads-down to grind out and condense as much information as I could as fast as I could.

I hit up "more iterators in action" - it was a great talk, and I am looking forward going back over the material. It was "information rich". After that was one of my favorite talks of the conference: Core Python Containers by Raymond D Hettinger - that talk was fantastic. I need to re-watch the video when it goes on "the youtube" to really absorb the content again, but it was really a great talk.

Then I hopped into the 2to3 talk - but then skipped out as I've been over that one before, and slipped into the OLPC testing talk by Titus (yes, I was the guy who cheered when he showed his kid on the screen). Titus is probably one of the best presenters PyCon has. He's engaging, funny and knows his stuff cold. Some of the tools he showed in the OLPC talk are really interesting for test automation, even if UI stuff is not my cup of tea, the back end stuff he's done with peekaboo is awesome.

Also, Titus' post on the "Death Spiral" is a must read. I liked my analogy (in the comments) that all Good Chefs (and cooks!) taste their food as they prepare and cook it to ensure it well, tastes good. The same applies to testing.

After that - it was fear. Yes, I was scared out of my mind to stand up the few hundred people left at the conference and do my lightning talk. According to people I trust, it was well received - and I got information out, which is great.

After that is was sprint-time, and a good time indeed.

PyCon this year was great, even if it did have it's "warts" with the sponsor stuff - it was also huge. I can't wait for next year. And thanks to everyone who asked about the kiddo and house. Right now, things are going well (albeit expensively with the house).

Onto the Miscellanea:

I stumbled onto this blog in the last few days, namely - this post. The author has made a few interesting posts on the (for lack of a better term) "pro static language" argument, here, here and here. It's an interesting read, even if I disagree with some of what the author says. Both static and dynamic (read: Duck/Runtime) typed languages have their pros and cons.

Speaking of Types, I found this article "What to know before debating type systems" quite good.

In Non-Typing news, this post (and this one) on Python Decorators is quite good, although more recently I saw some code that looked like this:

@foo
@bar(...)
@baz
@what
@yourekiddingmeright
def func():
insert lollerskates

Which made me realize, even among consenting adults, there's some people who drink way too much and code.

Which brings us to my final tidbit: Concurrency. There was some hubbub around the new Fork/Join stuff coming to the Java 7 JRE, the stuff outlining what's coming (including the original paper) is a series of "good reads".

I'd like to carve some time (yeah. ok.) out to think about a concurrency library similar in nature to java.util.concurrent for Python. I think it would be a useful addition, especially with the hopeful prospect of safethreading coming to town.

Back to my articles and other associated work.

Ivan Krstić resigns from OLPC

(Via Ars) This is surprising given how much he's dedicated to the OLPC project. If this is a sign of things to come, I guess it's another marker for "bad management can bring down any project".

I missed the post on Ivan's own blog - I blame pycon-haze. I guess this may help "explain" why his keynote was "about OLPC" and not "on behalf of". It's saddening to see someone who believed in the project as strongly as he did brought low in this fashion.

A followup on Concurrency within Python

Coming out of PyCon 2008 my head is again filled with thoughts on concurrency based on the talks and work I was involved with at the Con.

First up: I want to point out something which bothered me a little. Namely, that many people seemed resigned and pushing towards Jython/IronPython as the "concurrency answer". There was a lot of discussion/buzz about the fact that both of these implementations are the future of concurrency for python as a language due to the obvious lack of a GIL.

In discussing this with many people, I may have come off as defensive if nothing more than for the fact I constantly refuted that as "the answer". Don't get me wrong, I am eager, no, very eager to get my hands on the latest version of Jython to reach in and try out the java.util.concurrent package via Jython, I think access to that library will be great, for those working in a hybrid Java/Python shop.

But I don't think "Python's" future should be tightly coupled with the implementation of a runtime on top of another runtime. I think work has to be performed on the CPython interpreter to make it a viable contender and solution within the "concurrency" space. Python's strength will be found in multiple strong implementations of the interpreter.

Let me be clear: I believe Python programmers want to write Python - not Java, nor .Net. What attracts us to Python is a clean syntax. If I want concurrency in Python, I don't want to have to call into java.util.concurrent. Obviously, the Jython guys are not going to force this - threading will simply use Java threading, but I think a "pythonic" abstraction of the java.util.concurrent package as a whole would also be desired - so you could really use the power of the libraries, but stay wrapped in the warm pastry of Python.

Concurrency and distributed design is a large enough space that the use-cases and problems people want and need to address open the playing field for not just "one" solution (i.e: getting rid of the GIL) but a series of language and standard library improvements to Python as a whole.

My lightning talk touched on this: I think there are existing projects that are well-suited for inclusion into the standard library, and I believe Adam Olsen's work on the safe threading project (update here) will also help pave the way for a much more appealing future for CPython - even if it is in Py3k.

Python has a history of picking the best ideas from other languages and improving on them. For example: Python probably does not want to simply re-create the java.util.concurrent package - the concurrent package views threads as the solution. Python would be well suited to support "real threads" in as simple and an abstracted way as possible (a leaky abstraction at that).

Erlang has the Actor/Asynchonus Message passing model (a functional language implementation) (for more on Erlang concurrency, see this). While people may be envious of what they see as "perfect side effect free concurrency" - I doubt they envy the syntax or complex nature of it. Still yet - there are existing fork+exec+message passing libraries available for python right now that sidestep the GIL, but allow you to keep basic threading primitives and semantics and parallelize to cores, and clusters.

In discussion all of this with people at PyCon, I really began to realize that many people in the community look in envy to other language's concurrency implementations - including Java's Threads (even though many people still insist threads are impossible to get right).

I also realized that a lot of people are trying to solve a lot of various problems. When people hear distributed, concurrency and parallelism, they immediately bring up large data crunching (ala map reduce and its kin), distributed filesystems (ala GFS and Hadoop) to simple examples of jobs spread across multiple cores for math crunching and jobs passed to entire clusters of hundreds of machines.

Everyone is looking for an answer to their problem - and Python can not implement something that will address your problem. It can only provide the platform with the tools to allow you to solve it yourself.

I suggested the following to multiple people at pycon - this is what I view as the "cleanest" approach to providing the abstractions and tools people will need to move forward with CPython in this space.

1. Adopt/move forward with a version of CPython based on the work/concepts offered in Adam Olsen's Safe/Without GIL Threading work. Adopt the monitor and deadlock work for all of CPython, but leave the GIL removal as a compile-time option (python3.0 and python3.0-mt binaries).
2. Add the pyProcessing, pprocess, or Parallel-Python module to the standard library for those who want to use vanilla cpython, and to whom the fork method works. Which module is added should use existing Python semantics, but be powerful enough for advance/distributed usage. Note, I have started work on a PEP to get the pyProcessing module into the stdlib.
3. Add an Actor - or Actor-Like module to the functools module, in keeping with the concepts of the current implementations. Python's implementation would be obviously leaky. (Note, I am not a functional programmer by any stretch, I like my OOP, suggestions welcome). This could easily take advantage of both the Safe Threading work.
4. Add a lightweight (not XML) messaging system to couple with the Actor implementation . (no, I have no suggestions here)

Obviously, a varied approach like this doesn't conflict with the "one way to do it right" methodology of Python, in a problem space as varied as this - "one way to do it right" is completely dependent on the problem being solved. It can also help couch the discussion of what the problem they're trying to solve really is.

I've obviously been trying to help with the safe threading work - and I am writing a PEP for the inclusion of pyProcessing into the stdlib, but even this work is not the end - it's simply a start to help give people some batteries to plug into their concurrent toy car.

We should think about the building blocks to help people solve whatever problem that arises - we can't add a solution that fits any one problem, we can only provide the tools with which to build a great big concurrent future.

Now I have to go - my daughter has acquired the lock.

Also, thanks to Adam for helping review this!

Concurrency Lightning talk slides.

Here are the slides from my lightning talk for anyone interested.

Saturday @ Pycon

So, saturday has come and gone - most of it found me heads-down working with Rhamphoryncus (Adam) on the python-safethread(freethreading) project trying (and succeeding) to get the code ported to my OS/X box. Good times.

Yesterday the keynotes were fairly interesting - Glyph and Duncan got up and made a great Twisted announcement, namely that twisted is now a recognized non profit software foundation (Twisted Foundation). That's great news - and they also announced their first sponsors as well. Congratulations to the twisted guys.

After that, there was an amusing, and very good talk by Brian Fitzpatrick called "You *can* fool all of your users all of the time". In it, he outlined a lot of great points about listening to your users, customer service, and putting the users first.

I think he had a lot of good points that get forgotten by a lot of people in software - he also outlined the tenants which have helped make google successful (Putting your users first, speed matters, etc.). He really is a fantastic speaker, and I hope people keep his points in mind.

One of the best points he had was that all abstractions leak - and many people spend their time trying to close off those leaks. He countered with "keep it leaky" - the reason being, is that you make an abstraction to keep it simple for some users, but keeping it leaky means that users with more technical aptitude can then use it in m ore advanced ways. The example he gave was the google search page - most users type a string in, advanced users use statements, formulas, etc.

The second (3rd?) Keynote was done by Van Lindberg, a lawyer primarily focused on IP. In it, he outlined some less than popular views on Intellectual Property. I really appreciated his outline of how it came into being, the economics of knowledge outline, etc.

I can say that a lot of people were grumbling to many of his points - a good chunk of the people in the room are IP absolutists, leaning towards the more RMS view of IP in totality. I really did enjoy it, and I thought he raised some interesting food for thought for many people.

Admittedly, the patent system in the us is broken - no one disputes that, but knowledge does in fact, have a cost to create, and like any creation, people will want to be able to use that knowledge to gain wealth. This concept is not exclusive to the concept of open source/etc.

But that's all I'll say about that. I think Van was a decent speaker, he stumbled at times, but his presentation was filled with good information.

I ended up having to leave early to run to an open space talk on Multicore programming with Bruce Eckel/Ted Leung and others. Catching the tail end of the discussion, they were primarily focused on the Actor Model and how to use it within Python/how it's implemented in other languages. A lot of the concurrency discussion at the conference has been primarily focused on alternative interpreters - namely Jython, PyPy and IronPython.

Given I only caught the tail end, I suggested we have another discussion on concurrency later in the day (more on that later).

After that, I slipped in to catch the middle/end of the "Decorated State Machines" talk - I missed the intro, so coming into it late, given the density of the information, made it incredibly difficult to grok what was going on.

So, I skipped over to "Managing Complexity (and testing)" which was Fantastic - Matt Harrison did an excellent job outlining key concepts for code coverage, code complexity and designing/testing properly. He also outlined the key measurements used in code coverage and how best to apply them - I really learned a lot from his presentation. He's uploaded his slides here.

Next up, the ever effervescent Alex Martelli's talk on "Callback Patterns and Idioms in Python" - again, another fantastic, information rich talk. I can't say enough about this talk too. From a technical bent it was spot on and didn't bog down at all. You can get his slides from here.

The next hour or so was a blur - why? I was heads-down working on the free threading stuff.

So, the next time I came up for air was for the py.test talk, which was quite interesting, and I'm going to followup on using it + nose for some internal work. A lot of great information, if not a rough presentation.

I dropped out of the lightning talks to attend the open space discussion on concurrency - a lot of people showed up, and a lot of good ideas were talked about. I took some time to pimp processing.py and the free threading work. It definitely does seem like there is an itch in the community to scratch.

The biggest thing I can say to that though, is that people need to realize that simply dismissing threading, shared memory and existing methodologies to building parallel/concurrent applications as "wrong headed" is a bad idea.

The last thing we want as a community to simply dismiss these - instead we should approach the problem with a multi-homes approach. Work on fixing threading, support the Actor model and look into a functional and OO method of concurrent programming. (More on this later).

Lots of thoughts about this really, and I, as well as others need to realize there are many, many problems in this space looking for solutions, and no one solution/tool will address all of them.

And, I got the free threading stuff compiled on OS/X. Woop.

Interesting PyCon perspective from Bruce Eckel

Bruce Eckel has posted a brutally honest perspective on the overwhelming vendor exposure at Pycon this year.
I actually agree with many of Bruce's points - PyCon this year very obviously had commercial sponsorship, from the "diamond keynotes" to the Lightning talks. I did miss the lightning talks of old. Much like Bruce, I did find an oasis - the open space discussions (but that's because the ones I went to were on concurrency).

On the flip side, I enjoyed a great many of the talks, although there was some very painful ones where they were too dense, or the presenter just simply did know how to present.

My experience has been positive, and I have met some people, had some great discussions and learned some good stuff. I know this is simply a bump in the road until we hit a stride with PyCon as a regular con as a whole.

Commercial sponsorship is great, as long as it's clear, and not invasive. I expect to have vendors, recruiters and other stuff like that at the con - it's a programming con, if I didn't have recruiters recruiting, I'd assume we were doing something wrong.

Pycon: Friday Wrap up

Sadly, this wrap up will not include grilled vegetables, as the sandwiches for lunch did.

Of course I first hit up the keynote as every one else did - the keynote from White Oak Technologies was ok - they primarily outlined why Python made them successful, and what didn't go well for them including:

• Not enough people know python
• Few organization they consult to have python solutions
• Python "weirdness" is a turn off to some
• Python is slow

Chris took each one of the straw men down in subsequent slides - they used a lack of developer who knew python to differentiate between candidates - best quote "No one uses python 'by accident'". They also noted that on various searches for python expertise, there were very few out there in 2001 - but those that were out there were very high quality.

As for the lack of solutions: They turned it into an advocacy platform, and used it (taking a karma hit) to lock out competitors. Chris was quick to point out the latter is bad - but given a lot of the other firms they compete with immediately wanted to go in and re-write any python they found - customers are notoriously skittish about re-writes.

He also mentioned the fact that most of their clients are interested in the fish - not the fishing pole (tool versus output).

As for the weirdness, it's largely first impressions based on personal preferences, and those that aren't (no private variables, programming errors mainly found at runtime) are not deal breakers. In fact, in the case of the runtime errors brought forth by the latent typing issue - it forced them to be more rigorous in testing.

Finally, for python being slow - it keeps the disciplined in implementation. Some problems simply don't it a pure python solution, and picking the right tool for the job is a critical skill.

Overall, like I said - it was Ok. White Oak technologies can be found here.

Now, the next keynote was obviously the one everyone wants to know about - namely Guido's talk on Python 3000. Rather than try o do my own recap, Steve Holden did a fantastic summary of the GuidoNote.

I'm still excited by Python 3000, and the slow transition plan (in years, not months) including a python 2.6 and probably a 2.7 release should assuage some people fears of serious community shear.

After that - it was the talk free-for-all. I tried to consume (and failed) the talk by Travis Oliphant on the new buffer interface planned for python 3k - it's so far out of my current domain (and skills) that I simply could not drink from that fount.

I dropped in on "Database development with Jython, SQLAlchemy, and Hibernate" - the biggest take away from that being - Jython makes calls into java code such as JNDI and JDBC a breeze, and pythonic. I really do think based off Sun's new found love for Jython, it will become a very serious force in the community shortly. All in all, it was a good talk. (I also enjoyed the iPhone interference in the A/V equipment.)

Invariably - I hit up Brett Cannon's talk on "How import does it's thing" - without sucking up, I can honestly say it was a very clear, and concise overview of exactly what import does. I know I'm not alone on that based on hallway conversations. It was a great overview, and Brett's one of the better presenters here so far (in my opinion).

After that was Dr. Tim Couper's talk: "Python references and practical solutions to reference-related problems". So far, this is on the top of all the talks I've been too. The presentation was amusing, and rich in content. It was an excellent overview of how reference accounting is done in python and how to use weakrefs/etc. It was really a fantastic talk, I hope they upload the videos of these things someplace like youtube.

Then there was the "PyTriton: building a petabyte storage system" by Jonathan Ellis. I think he's a good presenter, but it was rushed due to the short time slot. Sadly I can't go into my thoughts on this one - it crosses the streams between PythonMe and WorkMe, I do have to say though - he doesn't pull any punches when he has an opinion (see: threads) which I appreciate.

I followed that up with Brian Warner's talk on "Tahoe: A Robust Distributed Secure Filesystem" - again, I can't cross the streams too much here, but I was sad-faced due to the lack of Python in the talk. It focused on the application, not how python was used in it.

Next up was supposed to be "Like Switching on the Light: Managing an Elastic Compute Cluster with Python", but honestly the presentation was poor, so I ducked out and switched to the High performance Network IO with Python + Libevent " talk, which was really good, the Pyevent library looks quite compelling for network apps. I need to find the rest of my notes sadly.

I'd wrap up with the lightning talks (which are always fun) but I mainly jotted down a series of URLs to followup on later.

Later on it was pizza, and blessed bed. Altogether a pretty good day - and a lot of great information and discussions with people in the halls and rooms. The sponsor/vendor rooms is hopping, and the recruiters are out en masse.

Hopefully today I'll be able to meet even more people!

Banging around pycon - say hello if you see me!

I'm wearing a t-shirt so bright and lime green that I'm almost positive you can see if from space.

Feel free to say hello if you see me. I, and many others, will be blogging throughout the day.

(funny note, due to the preponderance of men at the conference, one of the women's bathrooms was converted to a mens room. Does it stay taboo?)