jessenoller.com

An itch: Decent test case tracking/registration

Something that's been a thorn in my side for some time has been a distinct lack of open, flexible and rational test case tracking tools. Many testing groups simply use spreadsheets to track thing (this doesn't scale) and still others use tools that are proprietary/don't handle automated tests/are overly static.

Driving into work today/in the shower (where all good ideas come from) I got the itch again to "solve" this. The idea being to create a tool which:

• Is open source. Ideally, it's built with as many from-the-community tools as possible, and integrates with environments/tools like py.test, nose, trac, roundup, etc.
• Uses open/standard APIs for management. This is a key point for me, given the fact that I am primarily concerned with tracking automated test cases. I do not want to have to go an manually enter a new automated unit/functional test: I want to write something like a nose plugin which builds on the spec plugin by Titus brown to "register" a new test with the system. Manual test cases can still be managed via a decent UI - or better yet, uploaded/managed via CSVs or the uploading of "test scripts". A script being something which defines the test tags (areas), attributes (regression, etc) and the steps to achieve the test - all of this can be expressed in pseudo-code and pulled in. Automated tests get the same treatment via the doc strings/etc. I also want the APIs to be cross-language - there is no reason in this day and age to lock into one language.
• Is simple - this is critical. I want something easy for users to customize for various workflows, and I don't want a painful deployment. I want all data stored in an "open" format so that if users want - they can easily swap the system out.
• Allows users to swap around "pluggable" components to change behavior ala Pinax - think a pluggable CMS system for tracking test data.
• Allows for visualization ala junit output and charting.
• Easy to integrate into a "test execution manager" - the program running automated tests should be able to easily grok the information from the tracking/management app to decide "what tests to run".

I know there are tracking tools out there: I've used most of them, and I still have yet to find one that meets these criteria. They either try to force you into a single paradigm for test execution, or they don't support something as simple as a remote API. And most of the "big company" tools are highly proprietary.

Something which allows for automated unit/functional/regression tests to auto-register is key. Manually tracking and input of automated tests - especially when you start having hundreds of automated tests being developed sucks.

Not to mention - if you're the project manager - you want to know about all of the tests in the system - you don't want to have to go slogging through code, nor do you want to have check multiple data sources for information about what you have. You need a centralized repository.

Finally - yes, a more critical number when thinking about tests is the amount of coverage on the product. But when you start writing more macro functional tests, coverage is a hard thing to measure. You have to constantly consider refactoring/deleting old tests that may not have as much worth as the same test+a lot of other activity. Without some centralized place to track all of your tests (automated and manual) and the actions they perform it's hard to start that process. Wouldn't it be nice to be able to query all of your tests to find the ones with the common attribute "writes file to x" and then be able to deprecate the tests which just do that, but keep the ones that perform that action (which is a simple one) in the context of a larger test? This helps prevent regression test bloat: You always want better tests - but you don't want massive amounts of redundant/overly simplistic tests.

Thinking about it - it wouldn't be hard to develop and application in Django which accomplished most of these goals. The hard part (at least initially) is designing the initial database layout. With newforms, you can easily prototype a system which brings the "spreadsheet" paradigm into a web UI to get started for manual entry. A nose plugin to call an XMLRPC or other API is also insanely simple to implement.

I have a lot itches it seems. I need a cream or something.

ps: Yes, I know about the TestCaseManagement plugin for trac - I like the fact it really just front ends subversion, and allows for test case versioning. It would be entirely possible to have something which "specs" the automated tests and generates the XML, or to extend the plugin to parse the data it wants in standalone XML from automated tests. The problem is that you don't "assign" automated tests to people. The plugin is more aimed at manual tracking/execution.

The 2.6/3.0 Multiprocessing todo list

With the latest beta's out in the wild - and the test suite being 100% less locky with the fix for issue874900 - it's time to button down and start polishing/closing out the remaining bugs. Unfortunately, some of them are, well - "complicated".

This is more of a personal tracking for what's remaining - anyone who wants to pitch in and help is more than welcome (in fact, it's encouraged). Working on this stuff - like most of the python-core team is a part time job for me, so the more eyes the merrier.

• issue3256: Multiprocessing docs are not 3.0-ready
  • I need to basically review/apply the patch and run through the examples on both 2.6 and 3.0. Andrii rocks.
• issue874900:threading module can deadlock after fork
  • This bug isn't in my camp, but I want to see it to completion as it very obviously directly impacted the mp package.
• issue3419: multiprocessing module is racy
  • The connection refused error has been resolved, and I need to review Mark's proposed patch to the incref part of the bug
• issue3321:_multiprocessing.Connection() doesn't check handle
  • This is from Victor's fuzzing of the mp module. Still need to review/test - I attached his test updates to the bug today.
  • Need a windows-happy fix.
• issue3311:block operation on closed socket/pipe for multiprocessing
  • Another fuzzing bug courtesy of Victor.
• issue3352 : Deficiencies in multiprocessing/threading API
  • Work has started on this one courtesy of Ben Peterson - I am going to start focusing on this as well.
• issue3125: test_multiprocessing causes test_ctypes to fail
  • This one is done, pending a merge of the fix to py3k - see issue3385
• issue3270: test_multiprocessing: test_listener_client flakiness
  • Just pending an added doc warning for the 0.0.0.0 issue trent brought up.
• issue3350 : multiprocessing adds built-in types to the global copyreg.dispatch_table
  • Can probably be closed with the fix to issue3125
• issue3206:Multiprocessing Array and sharedctypes.Array error in docs/implementation
  • Doc issue, patch pending - I need to review. Andrii again, rocks.
• issue3111 : multiprocessing ppc Debian/ ia64 Ubuntu compilation error
  • I added a check to simply not run the tests that were failing on the platforms which raise an exception when creating the RLock - not the best, or final solution. Need to revisit.
• issue3110 : Multiprocessing package build problem on Solaris 10
  • Solaris build problem: I lack a solaris machine to test with - and my solaris-fu is weak sauce.
• issue3149 : multiprocessing build fails on Solaris 10
  • See above. With my solaris being weak, the most I can hope for is grabbing a VMWare Fusion image for Solaris 10 and learning the hard way.
• issue3578: 'dictionary changed size' error in test_multiprocessing
  • Antoine may have found the issue - need to research
• issue3518: multiprocessing: BaseManager.from_address documented but doesn't exist
  • Doc issue

nose-testconfig: released.

A couple of weeks ago I shot an email to the testing-in-python list asking if anyone knew of a nose plugin that could pass test-specific configuration data to the tests themselves.

No one had anything - but in true open source spirit, I was volunteered to satisfy my own request.

What I wanted was something simple, fast, and flexible. In fact, I wanted it so flexible as to be completely insecure and dangerous. I wanted to be able to support ConfigParser, YAML and pure Python configuration files.

One of the future goals is to have the configuration file data generated by some management layer responsible for coordinating nose test execution - so something flexible leaves my options open.

I wanted to avoid having a configuration module too out of band of the test execution system itself, but flexible enough to support a variety of data sources. It would be simple to auto-generate configuration from a basic YAML file and a database for example.

So, I hacked a plugin together - I spent most of my time learning a lot about nose internals and mucking around with various configuration languages.

Today, I finally got to upload the fruit of my loom (ha.) to PyPi - behold: nose-testconfig

It's at version 0.3 because I burned a version on learning the magical properties of PyPi uploading. Why my ReST is not parsing for the documentation, I don't know.

Right now, it works - it parses what it should parse, extends nose and I can use it in my tests. This smells like victory! Fear not though! I do plan on adding features/bugfixes as I find them or, better yet, if people point them out/request things.

p.s.: No it's not pep 8 compliant right now. Also, YAML is awesome.

Edit: I've uploaded the code to google-code. I considered launchpad, as I want to try out bzr, but launchpad actually suffers from too many features for something only 104 lines long.

Welcome to Parenthood, here’s your germs

For the past year I've had the exciting opportunity to be a father - it really is fantastic, especially now that new parts of her personality is coming through and she's in the "crazy aboriginal where are your pants" stage.

However, on the flip side, it's a lot of work - tiring work. It also comes with a side-effect. Disease. I've been sicker the past year than I ever have been in my life. I think I end up with a cold or something else every month. I've had sinus infections, colds, Hand-Foot-and-Mouth and lack-of-sleep induced insanity.

Good times. At least I got to see the new batman movie, which was amazing. I'm going to go blow my nose again.

Python 2.6 and 3.0 Beta 2 released!

The second beta release of Python 2.6 and the second beta release of Python 3.0 are good to go.

From here on out, we're working on stability and bug fixes for final - this means I've got to close out the remaining multiprocessing package bugs and clean up the docs.

I really encourage people to download and test - it's important that these releases are as high-quality as possible. There's a lot of changes in both 2.6 and 3.0 and there's a lot of ground to cover.

Fuzzing Python

Well, now that I have a moment to catch my breath - here's an interesting utility that was released late last week. It's a Fuzz testing framework (see wikipedia) written in Python called fusil.

The reason I know about it, is that the author sent an announcement (and a handful of bugs) to python-dev. He (he being Victor) wrote a plugin capable of fuzzing the CPython code/library. Most of the bugs he found were segmentation faults (you can see the full list here).

In any case, fuzzing is an area I've been meaning to get "more into" - I think it's awesome that A: the tool is in python and B: there's a plugin to test python-c-code. You can view the list of python modules tested here.

This is pretty awesome!