Skip to content

An itch: Decent test case tracking/registration

Something that’s been a thorn in my side for some time has been a distinct lack of open, flexible and rational test case tracking tools. Many testing groups simply use spreadsheets to track thing (this doesn’t scale) and still others use tools that are proprietary/don’t handle automated tests/are overly static.

Driving into work today/in the shower (where all good ideas come from) I got the itch again to “solve” this. The idea being to create a tool which:

  • Is open source. Ideally, it’s built with as many from-the-community tools as possible, and integrates with environments/tools like py.test, nose, trac, roundup, etc.
  • Uses open/standard APIs for management. This is a key point for me, given the fact that I am primarily concerned with tracking automated test cases. I do not want to have to go an manually enter a new automated unit/functional test: I want to write something like a nose plugin which builds on the spec plugin by Titus brown to “register” a new test with the system. Manual test cases can still be managed via a decent UI – or better yet, uploaded/managed via CSVs or the uploading of “test scripts”. A script being something which defines the test tags (areas), attributes (regression, etc) and the steps to achieve the test – all of this can be expressed in pseudo-code and pulled in. Automated tests get the same treatment via the doc strings/etc. I also want the APIs to be cross-language – there is no reason in this day and age to lock into one language.
  • Is simple – this is critical. I want something easy for users to customize for various workflows, and I don’t want a painful deployment. I want all data stored in an “open” format so that if users want – they can easily swap the system out.
  • Allows users to swap around “pluggable” components to change behavior ala Pinax – think a pluggable CMS system for tracking test data.
  • Allows for visualization ala junit output and charting.
  • Easy to integrate into a “test execution manager” – the program running automated tests should be able to easily grok the information from the tracking/management app to decide “what tests to run”.

I know there are tracking tools out there: I’ve used most of them, and I still have yet to find one that meets these criteria. They either try to force you into a single paradigm for test execution, or they don’t support something as simple as a remote API. And most of the “big company” tools are highly proprietary.

Something which allows for automated unit/functional/regression tests to auto-register is key. Manually tracking and input of automated tests – especially when you start having hundreds of automated tests being developed sucks.

Not to mention – if you’re the project manager – you want to know about all of the tests in the system – you don’t want to have to go slogging through code, nor do you want to have check multiple data sources for information about what you have. You need a centralized repository.

Finally – yes, a more critical number when thinking about tests is the amount of coverage on the product. But when you start writing more macro functional tests, coverage is a hard thing to measure. You have to constantly consider refactoring/deleting old tests that may not have as much worth as the same test+a lot of other activity. Without some centralized place to track all of your tests (automated and manual) and the actions they perform it’s hard to start that process. Wouldn’t it be nice to be able to query all of your tests to find the ones with the common attribute “writes file to x” and then be able to deprecate the tests which just do that, but keep the ones that perform that action (which is a simple one) in the context of a larger test? This helps prevent regression test bloat: You always want better tests – but you don’t want massive amounts of redundant/overly simplistic tests.

Thinking about it – it wouldn’t be hard to develop and application in Django which accomplished most of these goals. The hard part (at least initially) is designing the initial database layout. With newforms, you can easily prototype a system which brings the “spreadsheet” paradigm into a web UI to get started for manual entry. A nose plugin to call an XMLRPC or other API is also insanely simple to implement.

I have a lot itches it seems. I need a cream or something.

ps: Yes, I know about the TestCaseManagement plugin for trac – I like the fact it really just front ends subversion, and allows for test case versioning. It would be entirely possible to have something which “specs” the automated tests and generates the XML, or to extend the plugin to parse the data it wants in standalone XML from automated tests. The problem is that you don’t “assign” automated tests to people. The plugin is more aimed at manual tracking/execution.

  • Doug Napoleone
    Join us in Pinax (#django-hotclub on freenode.net). We are planning (well I am at least) a CMS platform on top of Pinax the same way we already have a social networking platform.

    There has also been quite a but of talk about a issue tracking/project management platform.
    (We are currently using it for the development of Pinax, though I still pine for trac myself...)

    I have been thinking about this myself quite a bit, but other projects keep taking up my time. I have been rather vocal about being against a full project tracking platform in Pinax. This is because whenever it is mentioned it is with words like 'and all we need is X and we are done!' Having written systems like this in the past these comments scare me, and I fear going down a rat hole. Thus I have wanted people to focus on other things which we desperately need (like automated testing). With that said, I would love to see this happen. I just want it done right with the proper fore though.
  • Ah, see - I wasn't proposing adding it to pinax. Something like this needs to start small (and stay small ala nose) and then it can be hooked into something like pinax. This is also side-stepping the project-management aspect of the overarching problem - focus on one thing within "this idea" (test case versioning, management and tracking) and do it well.

    Integrate via plugins to other systems: True, some people won't want another specialized application inside of their environ - but you keep the core of the application clean and integrate-as-needed. That's what pinax is all about right? :)
  • Doug Napoleone
    Correct, that is what it is about, and what I was thinking of was exactly what you are talking about. A reusable django app (and maybe an app_plugin) for storing and visualization of the data.

    For the data communication I was thinking of using Google protocol buffers:
    http://code.google.com/apis/protocolbuffers/doc...

    (and using a protobuf<=>model mapper ala newforms)

    The cool thing is this would not limit things to django, once the .proto files are standardized. this could allow for integration with Trac (to map things to checkins, etc) The key is to get the .proto files spec'd out and just use the django app as a proof of concept initial client and data store (or even a GAE datastore with visualization).

    All this would dovetail into what is happening with the CMS and Project Management stuff going on in Pinax. Hence my request that you join the IRC discussion as you might find some people interested in implementing this stuff. ;-)
  • Ah - I thought you were assuming I wanted to push it into pinax :)

    It's interesting - for data visualization, Andrian H. demoed an interesting dynamic visualization app pycon before last - it automatically "understood" the models and allowed highly flexible "clicking around" within the models. I can't remember what he called it, but at the time I was wicked excited about it.

    As for this - using protocol buffers is an interesting idea, I could see it using the .proto files as storage, but exposing them to end-users is equivalent to forcing them to define XML files for test cases.

    Hmm, you gave me some more to think about. One of the things I thought about last night was back-ending the entire system with subversion or another SCM so you can piggy-back the versioning instead of implementing your own.
  • Doug Napoleone
    1. Visualization

    Adrian's app is now in the django core as a contrib app called 'databrowse', and that was exactly what I was thinking of.

    2. proto=schema

    On the .proto files I was thinking of defining a set of proto's which are needed for storing the information. proto files just describe a schema, and there are tools which generate the reader/writer/rpc framework in multiple languages for you. Would you really want a schema-per-test? How on earth would you even begin to manage that?

    the purpose of the .proto files is so that the user does not need to define the xml files (or schema), or write code for dealing with the parsing/sending/syncing of that data. You get the C++, Java, and python writing, parsing, local storage and tcp transport layers done for you. the hope being that multiple people/projects would develop backend stores for use with nose and/or other test frameworks.

    Of course this would require a standardized set of proto files which nose uses. proto files are not just for data, but allow for API definitions and even calling those API's in an RPC style.

    So define a standard API set, and the standard schemas for the data used in that API, and then generate the C++, Java, and python code (and api stubs). Get that figured out and the rest is just integration.

    I work in the Burlington area if you ever want to sit down at a white board.
  • 1. Thank you. I love that thing
    2. Sorry, I spaced for a minute - your suggestion makes sense - you obviously don't want a schema-per-testcase

    The one thing about the protos is that they are flexible and do allow you to push things into the system easily. The thing is, from a client perspective, you could also do the same thing with JSON/etc - we don't need a lot of performance, and $LANGUAGES already support it rather than adding in an extra dependency

    The main thing I am stuck on is how to design the *storage* mechanism for the back-end. The API is pretty easy as you say - you can use protocol buffers, XML, pickles, etc

    It's the back end storage of the data in an indexable/visualize-able way and then putting together the basic viewing UI.

    Hmm. More stuff to think about. I may take you up on it since I'm in Acton :)
  • Rob
  • I considered it - but the problem with it is that it's just a layer (and a thin one) on top of bugzilla - it makes sense if you already have invested your time and resources into bugzilla and are willing to pay the maintenance cost. With the plethora of project management/wiki/bug system now in the ecosystem - something as big and heavy as bugzilla isn't as attractive.

    Besides, a layer on top of an big application is not something I *personally* want. A small core implementation with plugin support/extensibility and the ability to integrate with other applications is key. If forced to make the choice though - given testopia support a JSON API and it's something I know, I'd go with it for test case management - but not bug tracking given difficulty in customizing it without large amounts of invested time.

    That also said - if I do get a chance to write something, I'll steal some of the ideas. :)
  • Titus Brown
    Spec was actually written by Michal Kwiatkowski!

    I have an idea for a similar piece of software. If I write it will you maintain? :) (j/k)
  • If it's in Django, sure! :) I'm going to start some back-of-the-napkin design and think about it some more. This is something I know I need *now* in the context of current work, so maybe I can steal some time, although I don't know what I would short at this point.

    Any additional thoughts?
blog comments powered by Disqus