jessenoller.com

What are your favorite nose plugins? How do you run Nose?

So, I am pondering going all-out with Nose, and I am wondering what plugins people find the most useful for it, and also how people are using it.

I see two aspects of nose/any test execution mechanism: Unit testing "native" (i.e: python code) and running tests that are more functional in nature (i.e: not testing python, but instead testing a web interface).

What are the features of nose you found the most useful?

Too bad I couldn't find a decent nose picking graphic for this one.

Python 2.6a3 and 3.0a5 released

Barry sent the email out last night that both Python 2.6a3 and 3.0a5 are released - these are the final alphas for both. I'd go and grab em while they're still hot off the presses... Provided you're not already sync'ing from svn/bzr/mercurial/wtf.

• Get 2.6: http://www.python.org/download/releases/2.6/
• Get 3.0: http://www.python.org/download/releases/3.0/

My Python maxes out my cpus…

... And it's just the unittest I wrote for an application I'm cooking up:

Python is consuming much in the way of processor time. I <3 the processing module. All the workers are doing is tossing integers from one queue to another (in, out, result) and doing random multiplication on those integers.

Delicious.

PEP 370 Per user site-packages directory - accepted.

Guido just shot out email to the dev list(s) announcing he has accepted Christian Helmes' Per user site-packages directory PEP. This is awesome as it will provide a lot of functionality for those of us who don't want to sully the system package tree with modules we install. It could also help simplify things like zc.buildout and virtualenv.py

For giggles, you should read the discussion thread - please ignore my less-than-caffenated contribution to it.

Pep Turds

I've finally been given the "gift of time" to work on the processing module PEP - when I was reviewing the text template, I ran across this:

- Leave the little Emacs turd at the end of this file alone,
including the formfeed character ("^L", or \f).

I should really switch to a turd-friendly editor. I think VIM or Textmate are scatologically disinclined.

Lazyweb question: Python video manipulation libraries?

So here's a question for the lazyweb - I'm looking into doing some command-line/scripted video manipulation and I am looking for pointers to current/well-documented modules/libraries to use.

Specifically - I am looking to be able to stream-edit movies programmatically for a variety of formats (one of which is quicktime for a personal project) - ideally I would be able to load the video file and edit the stream (insert bit from other videos, append videos, etc) inside of Python.

The simplest use-case I can think of is building a montage of various videos, for example - say I downloaded non-flv (or flv formatted) videos of funny-baby scenes from youtube and wanted to chain them together into one big funny-baby montage.

Anyone have any links/pointers - the ones I have glanced at are:

• VideoParser: Mainly just for header information/parsing
• Flashticle: For flash video manipulation - no documentation
• PyMedia: Dated, the last update is from 2006. Does not compile on OS/X

Ideally, the library would allow stream editing/manipulation and be cross-platform. I'm willing to sack the xplatform requirement if the library is really good.

Twisted concurrency with Bruce Eckel.

I saw this on the 'tubes today: Concurrency with Python, Twisted, and Flex by Bruce Eckel.

Ever since I read it, I've gotten that tickle again to tackle twisted. He offers a pretty elegant solution to the problem of chunking up the work and spreading it across CPUs.

The draw for me to his solution is the asynchronous way in which the results are dispatched/captured (I'm going to be doing a lot of async work soon). It's a pretty good mini-intro into the semi-unpenetrable fort of Twisted.

Personally, I would have probably approached this using the processing module ala (note, I assumed access to Bruce's methods):

 
from processing import Pool, TimeoutError
pool = Pool(detectCPUs())
 
# ... skipping building the worklist, in bruce's case it's $cores * steps
TASKS = [ step1, step2, step3 ]
 
results = [pool.apply_async(t for t in TASKS]
imap_it = pool.imap(TASKS)
imap_unordered_it = pool.imap_unordered(TASKS)
 
print 'Ordered results using pool.apply_async():'
for r in results:
    print '\t', r.get()
print
 

Note, of course - I lifted the above right from the documentation examples for the module. Yes - I know, it lacks the network-ability that Bruce's twisted-based solution has innately (which does make his solution quite nice). You could do the same with the processing Client/Listeners in theory, but twisted is doing all of the connection management for him under the covers.

Dangit Bruce gave me the twisted itch again.

And later I'll do a more complete solution with the processing module, and compare the speeds of both against one another. I prefer the single-module solution over the twisted.* approach - but I can be swayed. :)

Finding Python people is hard.

...and finding Quality Assurance people with Python experience (especially advanced Test Engineering skills) is even harder.

The white oak guys at pycon talked about how they used python experience as a measure of skill/intent. Personally, having spent the better half of the last five years looking for people with good Python skill sets, and now looking to build out a serious team - I'm almost always in the position of hiring on people who know "just enough" and teaching them Python.

Teaching python is easy - but not always what you want to end up doing when you're going fast in a startup (although, I've lost track of how many times I've done this).

Excuse me while I go grumble - either you teach QA people automation/test engineering, or you try to find a programmer who wants to learn/do test engineering and teaching them python. No - I don't want programmers to "do QA" - I want them to write code which proves the product.

It's a hard sell to both parties. You also don't want QA people who view QA as subservient to Development, and Developers who don't see QA as subservient to the same. I technically view QA as one discipline, Development as another, but Test Engineering as the Hybrid of the two - and you need a strong background in both. You are writing complex code to test and prove the product - code which can sometimes be as complex as the product itself.

I'm willing to teach anyone anything I know to help them - doing so helps me in the long term build a strong team. Just finding the right people is not-so-easy.

I said in an interview recently - when asked if I wanted to be a Software Engineer or a Software Test Engineer - that if I am writing code in either one of the roles, I don't see the difference. You're writing tools/code and trying to develop new/interesting things in both roles. If you are writing product code: you still have to write test code. If you write tools for testing: you're writing "product code" (that needs testing code). The same pattern of write-test-write-test applies no matter what code you're writing.

There's nothing "less" about writing code that tests a product, just as there is nothing "unique" about writing the product code itself. Both must be tested, both are code. Would I like to write more product code? Sure - but I have to also write the tests that prove it anyway.

Just as an addendum - Corey made a post, and Terry made a post too both discussing this issue. There's also an awesome comment from another test engineer on the reddit discussion here.

Hi, my name is Jesse and I abuse list comprehensions.

I just found myself being quite abusive.

I have a series of dict{key:[list]) style objects, and I wanted to unpack it, convert it to a string, etc. (yes, variable names have been changed to protect the innocent)

 
return ',',join(['{{whizbang|%s}}' % i['foo'] for i in [ i for i in myDict[keyname]]])
 

Now, this could only be better if I:

• Included a lambda
• Had another nested comprehension

Other than the fact that what I wrote above is next to untestable - it's also hard to understand unless well, you're me. In order to debug it, you'd have to pprint the inbound dict and slowly unravel (from left to right) the operations being done.

Luckily, there's no math there!

So, I "cleaned" it up:

 
whizlist = []
for item in [k for k in myDict[keyname]]:
    whizlist.append('{{whizbang|%s}}' % item['foo'])
return ','.join(whizlist)
 

Hmm, I still seem to be juggling chainsaws here. Let's go for the "easy" button:

 
whizlist = []
for item in myDict[keyname]:
	whizlist.append('{{whizbang |%s}}' % item['foo'])
return ','.join(whizlist)
 

I could argue that the first is "faster" (although I'd need to prove it) - but ultimately the final "un-optimized" version is more pythonic.

Miscellanea and a Pycon closeout

Since being back in "the real world" - things have been crazy. I'm working on a PEP (albeit slowly) and trying to ramp up on python core-development while simultaneously trying to work on the safetheading stuff and several PyMag articles.

Did I mention that I can't work on any of this during the day? *head explodes*

Oh, and there was Easter Dinner to attend to as well? I do so love cooking.

I envy the Googler's out there and the others who get an allotment of time to "contribute to open source" or explore other technical avenues for many reasons, the obvious of which is "having more time to do something else".

To boot, I've been compulsively saving things to read/catch up on in my newreader. I'm up to about 205 items to cycle through.

All the whining done - I never did get to "close out" PyCon.

The last two years I've gone, I never stayed long enough on Sunday to see or do anything useful. This year, I wanted to stay through the core sprint, but alas - I had to go back Monday.

I was at least able to finish out the con, see some great talks and do a lightning talk on concurrency stuff. I also got to sit through Brett's intro to core development (which was great by the way) and try to get up and hacking on core.

I unfortunately spaced out during most of the morning keynotes (except for the OLPC update) as the mozilla-world isn't my particular cup of Joe. In addition to that, I got tapped by Jacob Kaplan-Moss to do a lightning talk (they had an open slot) - I said yes, and promptly went heads-down to grind out and condense as much information as I could as fast as I could.

I hit up "more iterators in action" - it was a great talk, and I am looking forward going back over the material. It was "information rich". After that was one of my favorite talks of the conference: Core Python Containers by Raymond D Hettinger - that talk was fantastic. I need to re-watch the video when it goes on "the youtube" to really absorb the content again, but it was really a great talk.

Then I hopped into the 2to3 talk - but then skipped out as I've been over that one before, and slipped into the OLPC testing talk by Titus (yes, I was the guy who cheered when he showed his kid on the screen). Titus is probably one of the best presenters PyCon has. He's engaging, funny and knows his stuff cold. Some of the tools he showed in the OLPC talk are really interesting for test automation, even if UI stuff is not my cup of tea, the back end stuff he's done with peekaboo is awesome.

Also, Titus' post on the "Death Spiral" is a must read. I liked my analogy (in the comments) that all Good Chefs (and cooks!) taste their food as they prepare and cook it to ensure it well, tastes good. The same applies to testing.

After that - it was fear. Yes, I was scared out of my mind to stand up the few hundred people left at the conference and do my lightning talk. According to people I trust, it was well received - and I got information out, which is great.

After that is was sprint-time, and a good time indeed.

PyCon this year was great, even if it did have it's "warts" with the sponsor stuff - it was also huge. I can't wait for next year. And thanks to everyone who asked about the kiddo and house. Right now, things are going well (albeit expensively with the house).

Onto the Miscellanea:

I stumbled onto this blog in the last few days, namely - this post. The author has made a few interesting posts on the (for lack of a better term) "pro static language" argument, here, here and here. It's an interesting read, even if I disagree with some of what the author says. Both static and dynamic (read: Duck/Runtime) typed languages have their pros and cons.

Speaking of Types, I found this article "What to know before debating type systems" quite good.

In Non-Typing news, this post (and this one) on Python Decorators is quite good, although more recently I saw some code that looked like this:

@foo
@bar(...)
@baz
@what
@yourekiddingmeright
def func():
insert lollerskates

Which made me realize, even among consenting adults, there's some people who drink way too much and code.

Which brings us to my final tidbit: Concurrency. There was some hubbub around the new Fork/Join stuff coming to the Java 7 JRE, the stuff outlining what's coming (including the original paper) is a series of "good reads".

I'd like to carve some time (yeah. ok.) out to think about a concurrency library similar in nature to java.util.concurrent for Python. I think it would be a useful addition, especially with the hopeful prospect of safethreading coming to town.

Back to my articles and other associated work.