An itch: Decent test case tracking/registration

July 31st, 2008 § 12 comments

Some­thing that’s been a thorn in my side for some time has been a dis­tinct lack of open, flex­i­ble and ratio­nal test case track­ing tools. Many test­ing groups sim­ply use spread­sheets to track thing (this doesn’t scale) and still oth­ers use tools that are proprietary/don’t han­dle auto­mated tests/are overly static.

Dri­ving into work today/in the shower (where all good ideas come from) I got the itch again to “solve” this. The idea being to cre­ate a tool which:

  • Is open source. Ide­ally, it’s built with as many from-the-community tools as pos­si­ble, and inte­grates with environments/tools like py.test, nose, trac, roundup, etc.
  • Uses open/standard APIs for man­age­ment. This is a key point for me, given the fact that I am pri­mar­ily con­cerned with track­ing auto­mated test cases. I do not want to have to go an man­u­ally enter a new auto­mated unit/functional test: I want to write some­thing like a nose plu­gin which builds on the spec plu­gin by Titus brown to “reg­is­ter” a new test with the sys­tem. Man­ual test cases can still be man­aged via a decent UI — or bet­ter yet, uploaded/managed via CSVs or the upload­ing of “test scripts”. A script being some­thing which defines the test tags (areas), attrib­utes (regres­sion, etc) and the steps to achieve the test — all of this can be expressed in pseudo-code and pulled in. Auto­mated tests get the same treat­ment via the doc strings/etc. I also want the APIs to be cross-language — there is no rea­son in this day and age to lock into one language.
  • Is sim­ple — this is crit­i­cal. I want some­thing easy for users to cus­tomize for var­i­ous work­flows, and I don’t want a painful deploy­ment. I want all data stored in an “open” for­mat so that if users want — they can eas­ily swap the sys­tem out.
  • Allows users to swap around “plug­gable” com­po­nents to change behav­ior ala Pinax — think a plug­gable CMS sys­tem for track­ing test data.
  • Allows for visu­al­iza­tion ala junit out­put and charting.
  • Easy to inte­grate into a “test exe­cu­tion man­ager” — the pro­gram run­ning auto­mated tests should be able to eas­ily grok the infor­ma­tion from the tracking/management app to decide “what tests to run”.

I know there are track­ing tools out there: I’ve used most of them, and I still have yet to find one that meets these cri­te­ria. They either try to force you into a sin­gle par­a­digm for test exe­cu­tion, or they don’t sup­port some­thing as sim­ple as a remote API. And most of the “big com­pany” tools are highly proprietary.

Some­thing which allows for auto­mated unit/functional/regression tests to auto-register is key. Man­u­ally track­ing and input of auto­mated tests — espe­cially when you start hav­ing hun­dreds of auto­mated tests being devel­oped sucks.

Not to men­tion — if you’re the project man­ager — you want to know about all of the tests in the sys­tem — you don’t want to have to go slog­ging through code, nor do you want to have check mul­ti­ple data sources for infor­ma­tion about what you have. You need a cen­tral­ized repository.

Finally — yes, a more crit­i­cal num­ber when think­ing about tests is the amount of cov­er­age on the prod­uct. But when you start writ­ing more macro func­tional tests, cov­er­age is a hard thing to mea­sure. You have to con­stantly con­sider refactoring/deleting old tests that may not have as much worth as the same test+a lot of other activ­ity. With­out some cen­tral­ized place to track all of your tests (auto­mated and man­ual) and the actions they per­form it’s hard to start that process. Wouldn’t it be nice to be able to query all of your tests to find the ones with the com­mon attribute “writes file to x” and then be able to dep­re­cate the tests which just do that, but keep the ones that per­form that action (which is a sim­ple one) in the con­text of a larger test? This helps pre­vent regres­sion test bloat: You always want bet­ter tests — but you don’t want mas­sive amounts of redundant/overly sim­plis­tic tests.

Think­ing about it — it wouldn’t be hard to develop and appli­ca­tion in Django which accom­plished most of these goals. The hard part (at least ini­tially) is design­ing the ini­tial data­base lay­out. With new­forms, you can eas­ily pro­to­type a sys­tem which brings the “spread­sheet” par­a­digm into a web UI to get started for man­ual entry. A nose plu­gin to call an XMLRPC or other API is also insanely sim­ple to implement.

I have a lot itches it seems. I need a cream or something.

ps: Yes, I know about the Test­Case­M­an­age­ment plu­gin for trac — I like the fact it really just front ends sub­ver­sion, and allows for test case ver­sion­ing. It would be entirely pos­si­ble to have some­thing which “specs” the auto­mated tests and gen­er­ates the XML, or to extend the plu­gin to parse the data it wants in stand­alone XML from auto­mated tests. The prob­lem is that you don’t “assign” auto­mated tests to peo­ple. The plu­gin is more aimed at man­ual tracking/execution.

  • Titus Brown

    Spec was actu­ally writ­ten by Michal Kwiatkowski!

    I have an idea for a sim­i­lar piece of soft­ware. If I write it will you main­tain? :) (j/k)

  • jnoller

    If it’s in Django, sure! :) I’m going to start some back-of-the-napkin design and think about it some more. This is some­thing I know I need *now* in the con­text of cur­rent work, so maybe I can steal some time, although I don’t know what I would short at this point.

    Any addi­tional thoughts?

  • http://robs-blog.crickers.com/ Rob
  • Doug Napoleone

    Join us in Pinax (#django-hotclub on freenode.net). We are plan­ning (well I am at least) a CMS plat­form on top of Pinax the same way we already have a social net­work­ing platform.

    There has also been quite a but of talk about a issue tracking/project man­age­ment plat­form.
    (We are cur­rently using it for the devel­op­ment of Pinax, though I still pine for trac myself…)

    I have been think­ing about this myself quite a bit, but other projects keep tak­ing up my time. I have been rather vocal about being against a full project track­ing plat­form in Pinax. This is because when­ever it is men­tioned it is with words like ‘and all we need is X and we are done!’ Hav­ing writ­ten sys­tems like this in the past these com­ments scare me, and I fear going down a rat hole. Thus I have wanted peo­ple to focus on other things which we des­per­ately need (like auto­mated test­ing). With that said, I would love to see this hap­pen. I just want it done right with the proper fore though.

  • jnoller

    I con­sid­ered it — but the prob­lem with it is that it’s just a layer (and a thin one) on top of bugzilla — it makes sense if you already have invested your time and resources into bugzilla and are will­ing to pay the main­te­nance cost. With the plethora of project management/wiki/bug sys­tem now in the ecosys­tem — some­thing as big and heavy as bugzilla isn’t as attractive.

    Besides, a layer on top of an big appli­ca­tion is not some­thing I *per­son­ally* want. A small core imple­men­ta­tion with plu­gin support/extensibility and the abil­ity to inte­grate with other appli­ca­tions is key. If forced to make the choice though — given testopia sup­port a JSON API and it’s some­thing I know, I’d go with it for test case man­age­ment — but not bug track­ing given dif­fi­culty in cus­tomiz­ing it with­out large amounts of invested time.

    That also said — if I do get a chance to write some­thing, I’ll steal some of the ideas. :)

  • jnoller

    Ah, see — I wasn’t propos­ing adding it to pinax. Some­thing like this needs to start small (and stay small ala nose) and then it can be hooked into some­thing like pinax. This is also side-stepping the project-management aspect of the over­ar­ch­ing prob­lem — focus on one thing within “this idea” (test case ver­sion­ing, man­age­ment and track­ing) and do it well.

    Inte­grate via plu­g­ins to other sys­tems: True, some peo­ple won’t want another spe­cial­ized appli­ca­tion inside of their env­i­ron — but you keep the core of the appli­ca­tion clean and integrate-as-needed. That’s what pinax is all about right? :)

  • Doug Napoleone

    Cor­rect, that is what it is about, and what I was think­ing of was exactly what you are talk­ing about. A reusable django app (and maybe an app_plugin) for stor­ing and visu­al­iza­tion of the data.

    For the data com­mu­ni­ca­tion I was think­ing of using Google pro­to­col buffers:
    http://code.google.com/apis/protocolbuffers/doc…

    (and using a protobuf<=>model map­per ala newforms)

    The cool thing is this would not limit things to django, once the .proto files are stan­dard­ized. this could allow for inte­gra­tion with Trac (to map things to check­ins, etc) The key is to get the .proto files spec’d out and just use the django app as a proof of con­cept ini­tial client and data store (or even a GAE data­s­tore with visualization).

    All this would dove­tail into what is hap­pen­ing with the CMS and Project Man­age­ment stuff going on in Pinax. Hence my request that you join the IRC dis­cus­sion as you might find some peo­ple inter­ested in imple­ment­ing this stuff. ;-)

  • jnoller

    Ah — I thought you were assum­ing I wanted to push it into pinax :)

    It’s inter­est­ing — for data visu­al­iza­tion, Andrian H. demoed an inter­est­ing dynamic visu­al­iza­tion app pycon before last — it auto­mat­i­cally “under­stood” the mod­els and allowed highly flex­i­ble “click­ing around” within the mod­els. I can’t remem­ber what he called it, but at the time I was wicked excited about it.

    As for this — using pro­to­col buffers is an inter­est­ing idea, I could see it using the .proto files as stor­age, but expos­ing them to end-users is equiv­a­lent to forc­ing them to define XML files for test cases.

    Hmm, you gave me some more to think about. One of the things I thought about last night was back-ending the entire sys­tem with sub­ver­sion or another SCM so you can piggy-back the ver­sion­ing instead of imple­ment­ing your own.

  • Doug Napoleone

    1. Visu­al­iza­tion

    Adrian’s app is now in the django core as a con­trib app called ‘data­browse’, and that was exactly what I was think­ing of.

    2. proto=schema

    On the .proto files I was think­ing of defin­ing a set of proto’s which are needed for stor­ing the infor­ma­tion. proto files just describe a schema, and there are tools which gen­er­ate the reader/writer/rpc frame­work in mul­ti­ple lan­guages for you. Would you really want a schema-per-test? How on earth would you even begin to man­age that?

    the pur­pose of the .proto files is so that the user does not need to define the xml files (or schema), or write code for deal­ing with the parsing/sending/syncing of that data. You get the C++, Java, and python writ­ing, pars­ing, local stor­age and tcp trans­port lay­ers done for you. the hope being that mul­ti­ple people/projects would develop back­end stores for use with nose and/or other test frameworks.

    Of course this would require a stan­dard­ized set of proto files which nose uses. proto files are not just for data, but allow for API def­i­n­i­tions and even call­ing those API’s in an RPC style.

    So define a stan­dard API set, and the stan­dard schemas for the data used in that API, and then gen­er­ate the C++, Java, and python code (and api stubs). Get that fig­ured out and the rest is just integration.

    I work in the Burling­ton area if you ever want to sit down at a white board.

  • jnoller

    1. Thank you. I love that thing
    2. Sorry, I spaced for a minute — your sug­ges­tion makes sense — you obvi­ously don’t want a schema-per-testcase

    The one thing about the pro­tos is that they are flex­i­ble and do allow you to push things into the sys­tem eas­ily. The thing is, from a client per­spec­tive, you could also do the same thing with JSON/etc — we don’t need a lot of per­for­mance, and $LANGUAGES already sup­port it rather than adding in an extra dependency

    The main thing I am stuck on is how to design the *stor­age* mech­a­nism for the back-end. The API is pretty easy as you say — you can use pro­to­col buffers, XML, pick­les, etc

    It’s the back end stor­age of the data in an indexable/visualize-able way and then putting together the basic view­ing UI.

    Hmm. More stuff to think about. I may take you up on it since I’m in Acton :)

  • Doug Napoleone

    1. Visu­al­iza­tion

    Adrian’s app is now in the django core as a con­trib app called ‘data­browse’, and that was exactly what I was think­ing of.

    2. proto=schema

    On the .proto files I was think­ing of defin­ing a set of proto’s which are needed for stor­ing the infor­ma­tion. proto files just describe a schema, and there are tools which gen­er­ate the reader/writer/rpc frame­work in mul­ti­ple lan­guages for you. Would you really want a schema-per-test? How on earth would you even begin to man­age that?

    the pur­pose of the .proto files is so that the user does not need to define the xml files (or schema), or write code for deal­ing with the parsing/sending/syncing of that data. You get the C++, Java, and python writ­ing, pars­ing, local stor­age and tcp trans­port lay­ers done for you. the hope being that mul­ti­ple people/projects would develop back­end stores for use with nose and/or other test frameworks.

    Of course this would require a stan­dard­ized set of proto files which nose uses. proto files are not just for data, but allow for API def­i­n­i­tions and even call­ing those API’s in an RPC style.

    So define a stan­dard API set, and the stan­dard schemas for the data used in that API, and then gen­er­ate the C++, Java, and python code (and api stubs). Get that fig­ured out and the rest is just integration.

    I work in the Burling­ton area if you ever want to sit down at a white board.

  • http://jessenoller.com jnoller

    1. Thank you. I love that thing
    2. Sorry, I spaced for a minute — your sug­ges­tion makes sense — you obvi­ously don’t want a schema-per-testcase

    The one thing about the pro­tos is that they are flex­i­ble and do allow you to push things into the sys­tem eas­ily. The thing is, from a client per­spec­tive, you could also do the same thing with JSON/etc — we don’t need a lot of per­for­mance, and $LANGUAGES already sup­port it rather than adding in an extra dependency

    The main thing I am stuck on is how to design the *stor­age* mech­a­nism for the back-end. The API is pretty easy as you say — you can use pro­to­col buffers, XML, pick­les, etc

    It’s the back end stor­age of the data in an indexable/visualize-able way and then putting together the basic view­ing UI.

    Hmm. More stuff to think about. I may take you up on it since I’m in Acton :)

What's this?

You are currently reading An itch: Decent test case tracking/registration at jessenoller.com.

meta