jessenoller.com

What are your favorite nose plugins? How do you run Nose?

So, I am pondering going all-out with Nose, and I am wondering what plugins people find the most useful for it, and also how people are using it.

I see two aspects of nose/any test execution mechanism: Unit testing "native" (i.e: python code) and running tests that are more functional in nature (i.e: not testing python, but instead testing a web interface).

What are the features of nose you found the most useful?

Too bad I couldn't find a decent nose picking graphic for this one.

Python 2.6a3 and 3.0a5 released

Barry sent the email out last night that both Python 2.6a3 and 3.0a5 are released - these are the final alphas for both. I'd go and grab em while they're still hot off the presses... Provided you're not already sync'ing from svn/bzr/mercurial/wtf.

• Get 2.6: http://www.python.org/download/releases/2.6/
• Get 3.0: http://www.python.org/download/releases/3.0/

My Python maxes out my cpus…

... And it's just the unittest I wrote for an application I'm cooking up:

Python is consuming much in the way of processor time. I <3 the processing module. All the workers are doing is tossing integers from one queue to another (in, out, result) and doing random multiplication on those integers.

Delicious.

PEP 370 Per user site-packages directory - accepted.

Guido just shot out email to the dev list(s) announcing he has accepted Christian Helmes' Per user site-packages directory PEP. This is awesome as it will provide a lot of functionality for those of us who don't want to sully the system package tree with modules we install. It could also help simplify things like zc.buildout and virtualenv.py

For giggles, you should read the discussion thread - please ignore my less-than-caffenated contribution to it.

Pep Turds

I've finally been given the "gift of time" to work on the processing module PEP - when I was reviewing the text template, I ran across this:

- Leave the little Emacs turd at the end of this file alone,
including the formfeed character ("^L", or \f).

I should really switch to a turd-friendly editor. I think VIM or Textmate are scatologically disinclined.

Lazyweb question: Python video manipulation libraries?

So here's a question for the lazyweb - I'm looking into doing some command-line/scripted video manipulation and I am looking for pointers to current/well-documented modules/libraries to use.

Specifically - I am looking to be able to stream-edit movies programmatically for a variety of formats (one of which is quicktime for a personal project) - ideally I would be able to load the video file and edit the stream (insert bit from other videos, append videos, etc) inside of Python.

The simplest use-case I can think of is building a montage of various videos, for example - say I downloaded non-flv (or flv formatted) videos of funny-baby scenes from youtube and wanted to chain them together into one big funny-baby montage.

Anyone have any links/pointers - the ones I have glanced at are:

• VideoParser: Mainly just for header information/parsing
• Flashticle: For flash video manipulation - no documentation
• PyMedia: Dated, the last update is from 2006. Does not compile on OS/X

Ideally, the library would allow stream editing/manipulation and be cross-platform. I'm willing to sack the xplatform requirement if the library is really good.

Twisted concurrency with Bruce Eckel.

I saw this on the 'tubes today: Concurrency with Python, Twisted, and Flex by Bruce Eckel.

Ever since I read it, I've gotten that tickle again to tackle twisted. He offers a pretty elegant solution to the problem of chunking up the work and spreading it across CPUs.

The draw for me to his solution is the asynchronous way in which the results are dispatched/captured (I'm going to be doing a lot of async work soon). It's a pretty good mini-intro into the semi-unpenetrable fort of Twisted.

Personally, I would have probably approached this using the processing module ala (note, I assumed access to Bruce's methods):

 
from processing import Pool, TimeoutError
pool = Pool(detectCPUs())
 
# ... skipping building the worklist, in bruce's case it's $cores * steps
TASKS = [ step1, step2, step3 ]
 
results = [pool.apply_async(t for t in TASKS]
imap_it = pool.imap(TASKS)
imap_unordered_it = pool.imap_unordered(TASKS)
 
print 'Ordered results using pool.apply_async():'
for r in results:
    print '\t', r.get()
print
 

Note, of course - I lifted the above right from the documentation examples for the module. Yes - I know, it lacks the network-ability that Bruce's twisted-based solution has innately (which does make his solution quite nice). You could do the same with the processing Client/Listeners in theory, but twisted is doing all of the connection management for him under the covers.

Dangit Bruce gave me the twisted itch again.

And later I'll do a more complete solution with the processing module, and compare the speeds of both against one another. I prefer the single-module solution over the twisted.* approach - but I can be swayed. :)

Finding Python people is hard.

...and finding Quality Assurance people with Python experience (especially advanced Test Engineering skills) is even harder.

The white oak guys at pycon talked about how they used python experience as a measure of skill/intent. Personally, having spent the better half of the last five years looking for people with good Python skill sets, and now looking to build out a serious team - I'm almost always in the position of hiring on people who know "just enough" and teaching them Python.

Teaching python is easy - but not always what you want to end up doing when you're going fast in a startup (although, I've lost track of how many times I've done this).

Excuse me while I go grumble - either you teach QA people automation/test engineering, or you try to find a programmer who wants to learn/do test engineering and teaching them python. No - I don't want programmers to "do QA" - I want them to write code which proves the product.

It's a hard sell to both parties. You also don't want QA people who view QA as subservient to Development, and Developers who don't see QA as subservient to the same. I technically view QA as one discipline, Development as another, but Test Engineering as the Hybrid of the two - and you need a strong background in both. You are writing complex code to test and prove the product - code which can sometimes be as complex as the product itself.

I'm willing to teach anyone anything I know to help them - doing so helps me in the long term build a strong team. Just finding the right people is not-so-easy.

I said in an interview recently - when asked if I wanted to be a Software Engineer or a Software Test Engineer - that if I am writing code in either one of the roles, I don't see the difference. You're writing tools/code and trying to develop new/interesting things in both roles. If you are writing product code: you still have to write test code. If you write tools for testing: you're writing "product code" (that needs testing code). The same pattern of write-test-write-test applies no matter what code you're writing.

There's nothing "less" about writing code that tests a product, just as there is nothing "unique" about writing the product code itself. Both must be tested, both are code. Would I like to write more product code? Sure - but I have to also write the tests that prove it anyway.

Just as an addendum - Corey made a post, and Terry made a post too both discussing this issue. There's also an awesome comment from another test engineer on the reddit discussion here.

Breaking up is hard to do (because I got a new job)

Earlier this week - I made the rather difficult decision and resigned from my current job. Yes - I have a new one in the bag.

The job I resigned from has been one of the best, most rewarding jobs in my career. I've had the luxury of working with some of the best engineers I have ever met, on a problem/solution that is, by far, the best in it's class.

I've been working with what I count as my second family for close to five years. You don't walk away from something like that lightly, or without good reason.

My reason was simple: I enjoy startups. I love blazing trails, exploring technology. I love pushing boundaries and making something tangible and new where there was nothing before. It's why I love software.

Some people would say that software is not a painting - and to an extent, they are right. But the pride you feel when you see your software come alive, get adopted and grow day after day is much the same feeling of accomplishment and pride you get when you do finish a work of art.

Personally, I feel that startups engender the wild west - all ideas are open and ready to be explored, much of the territory is uncharted, and you remain (in many cases) unencumbered by processes, bureaucracy and history.

That all being said - it's not without an aching heart that I say good bye to my friends, colleagues and the product I helped usher into the world. Both the product, and the people remain best-in-class, and I wish both the company, and the people the best in the future.

The great (and sometimes bad) thing about startups though, is the fact that if you live in a place where startups are always coming and going (i.e: Boston) then sooner or later, you're going to end up working with many of the same people again. Due to this, I'm ultimately OK with saying good bye to many of the coworkers I have today, because I will see them again in due time.

That all being said - I have a new job (obviously). Later this month I will begin working for an Acton, MA startup called Blackwave. I've had the pleasure of meeting the people there and exploring the technology, and I have to say I got that old spark back in my head. The spark that keeps you up at night thinking about how to solve The Problem and how to test The Problem. The one that makes you get up early to write notes down to yourself like "if we simply change x here and here, then in theory we can cut our testing to y..." and so on. It's the spark of inspiration.

In speaking with them - I wanted to solidify something that's been chewing at me for a little bit - and that's having some amount of freedom to contribute to open source - in my case, Python-Core and python-related projects. I'm happy to say, Blackwave was more than willing to express their support and openness to contributing to the community. I can't go into details - but with any luck, I should be able to be a lot more open with much of my work, and also give a lot back.

Again - with that all being said - I will take this one opportunity to say - yes, Blackwave is hiring. If you're a python/java/c/c++ person, or you just love doing QA and testing and are looking for a small, fast moving team - and live in the Boston/Metrowest area (southern New Hampshire is OK) - go ahead and drop me an email. I'm definitely putting feelers out for any Pythonistas in the area who are looking around. I'm doubly interested in Pythonistas with a QA background to help focus on test engineering/automation.

Send your resumes to me (jnoller at gmail dot com)!

I'm looking forward to the upcoming months with excitement.

Hi, my name is Jesse and I abuse list comprehensions.

I just found myself being quite abusive.

I have a series of dict{key:[list]) style objects, and I wanted to unpack it, convert it to a string, etc. (yes, variable names have been changed to protect the innocent)

 
return ',',join(['{{whizbang|%s}}' % i['foo'] for i in [ i for i in myDict[keyname]]])
 

Now, this could only be better if I:

• Included a lambda
• Had another nested comprehension

Other than the fact that what I wrote above is next to untestable - it's also hard to understand unless well, you're me. In order to debug it, you'd have to pprint the inbound dict and slowly unravel (from left to right) the operations being done.

Luckily, there's no math there!

So, I "cleaned" it up:

 
whizlist = []
for item in [k for k in myDict[keyname]]:
    whizlist.append('{{whizbang|%s}}' % item['foo'])
return ','.join(whizlist)
 

Hmm, I still seem to be juggling chainsaws here. Let's go for the "easy" button:

 
whizlist = []
for item in myDict[keyname]:
	whizlist.append('{{whizbang |%s}}' % item['foo'])
return ','.join(whizlist)
 

I could argue that the first is "faster" (although I'd need to prove it) - but ultimately the final "un-optimized" version is more pythonic.