jessenoller.com

Some­thing that’s been a thorn in my side for some time has been a dis­tinct lack of open, flex­i­ble and ratio­nal test case track­ing tools. Many test­ing groups sim­ply use spread­sheets to track thing (this doesn’t scale) and still oth­ers use tools that are proprietary/don’t han­dle auto­mated tests/are overly static.

Dri­ving into work today/in the shower (where all good ideas come from) I got the itch again to “solve” this. The idea being to cre­ate a tool which:

• Is open source. Ide­ally, it’s built with as many from-the-community tools as pos­si­ble, and inte­grates with environments/tools like py.test, nose, trac, roundup, etc.
• Uses open/standard APIs for man­age­ment. This is a key point for me, given the fact that I am pri­mar­ily con­cerned with track­ing auto­mated test cases. I do not want to have to go an man­u­ally enter a new auto­mated unit/functional test: I want to write some­thing like a nose plu­gin which builds on the spec plu­gin by Titus brown to “reg­is­ter” a new test with the sys­tem. Man­ual test cases can still be man­aged via a decent UI — or bet­ter yet, uploaded/managed via CSVs or the upload­ing of “test scripts”. A script being some­thing which defines the test tags (areas), attrib­utes (regres­sion, etc) and the steps to achieve the test — all of this can be expressed in pseudo-code and pulled in. Auto­mated tests get the same treat­ment via the doc strings/etc. I also want the APIs to be cross-language — there is no rea­son in this day and age to lock into one language.
• Is sim­ple — this is crit­i­cal. I want some­thing easy for users to cus­tomize for var­i­ous work­flows, and I don’t want a painful deploy­ment. I want all data stored in an “open” for­mat so that if users want — they can eas­ily swap the sys­tem out.
• Allows users to swap around “plug­gable” com­po­nents to change behav­ior ala Pinax — think a plug­gable CMS sys­tem for track­ing test data.
• Allows for visu­al­iza­tion ala junit out­put and charting.
• Easy to inte­grate into a “test exe­cu­tion man­ager” — the pro­gram run­ning auto­mated tests should be able to eas­ily grok the infor­ma­tion from the tracking/management app to decide “what tests to run”.

I know there are track­ing tools out there: I’ve used most of them, and I still have yet to find one that meets these cri­te­ria. They either try to force you into a sin­gle par­a­digm for test exe­cu­tion, or they don’t sup­port some­thing as sim­ple as a remote API. And most of the “big com­pany” tools are highly proprietary.

Some­thing which allows for auto­mated unit/functional/regression tests to auto-register is key. Man­u­ally track­ing and input of auto­mated tests — espe­cially when you start hav­ing hun­dreds of auto­mated tests being devel­oped sucks.

Not to men­tion — if you’re the project man­ager — you want to know about all of the tests in the sys­tem — you don’t want to have to go slog­ging through code, nor do you want to have check mul­ti­ple data sources for infor­ma­tion about what you have. You need a cen­tral­ized repository.

Finally — yes, a more crit­i­cal num­ber when think­ing about tests is the amount of cov­er­age on the prod­uct. But when you start writ­ing more macro func­tional tests, cov­er­age is a hard thing to mea­sure. You have to con­stantly con­sider refactoring/deleting old tests that may not have as much worth as the same test+a lot of other activ­ity. With­out some cen­tral­ized place to track all of your tests (auto­mated and man­ual) and the actions they per­form it’s hard to start that process. Wouldn’t it be nice to be able to query all of your tests to find the ones with the com­mon attribute “writes file to x” and then be able to dep­re­cate the tests which just do that, but keep the ones that per­form that action (which is a sim­ple one) in the con­text of a larger test? This helps pre­vent regres­sion test bloat: You always want bet­ter tests — but you don’t want mas­sive amounts of redundant/overly sim­plis­tic tests.

Think­ing about it — it wouldn’t be hard to develop and appli­ca­tion in Django which accom­plished most of these goals. The hard part (at least ini­tially) is design­ing the ini­tial data­base lay­out. With new­forms, you can eas­ily pro­to­type a sys­tem which brings the “spread­sheet” par­a­digm into a web UI to get started for man­ual entry. A nose plu­gin to call an XMLRPC or other API is also insanely sim­ple to implement.

I have a lot itches it seems. I need a cream or something.

ps: Yes, I know about the Test­Case­M­an­age­ment plu­gin for trac — I like the fact it really just front ends sub­ver­sion, and allows for test case ver­sion­ing. It would be entirely pos­si­ble to have some­thing which “specs” the auto­mated tests and gen­er­ates the XML, or to extend the plu­gin to parse the data it wants in stand­alone XML from auto­mated tests. The prob­lem is that you don’t “assign” auto­mated tests to peo­ple. The plu­gin is more aimed at man­ual tracking/execution.

Something that's been a thorn in my side for some time has been a distinct lack of open, flexible and rational test case tracking tools. Many testing groups simply use spreadsheets to track thing (this doesn't scale) and still others use tools that are proprietary/don't handle automated tests/are overly static. Driving into work today/in the ...

So, I am pon­der­ing going all-out with Nose, and I am won­der­ing what plu­g­ins peo­ple find the most use­ful for it, and also how peo­ple are using it.

I see two aspects of nose/any test exe­cu­tion mech­a­nism: Unit test­ing “native” (i.e: python code) and run­ning tests that are more func­tional in nature (i.e: not test­ing python, but instead test­ing a web interface).

What are the fea­tures of nose you found the most useful?

Too bad I couldn’t find a decent nose pick­ing graphic for this one.

So, I am pondering going all-out with Nose, and I am wondering what plugins people find the most useful for it, and also how people are using it. I see two aspects of nose/any test execution mechanism: Unit testing "native" (i.e: python code) and running tests that are more functional in nature (i.e: not testing ...

One machine can do the work of fifty ordi­nary men. No machine can do the work of one extra­or­di­nary man. –Elbert Hubbard

I find this to be very, very true when I talk to peo­ple about 100% automa­tion goals, and out­sourc­ing of iterative/isolated tasks.

Peo­ple always respond “automation/outsourcing will take my job!”. My reply is always the same — no, it won’t, not if you alter the collateral/value you gen­er­ate. Peo­ple have the capac­ity to grow, to change and to learn. The automa­tion of repeat­able steps, and the out­sourc­ing of iso­lated projects (gen­er­ally, iter­a­tive and sim­ple) frees peo­ple to excel in their area, to become domain spe­cial­ists, and to gen­er­ally “become better”.

Now, does it always work this way? No. Large cor­po­ra­tions are noto­ri­ous for automating/outsourcing and then can­ning a large swath of peo­ple. Like all ideas, and like all tools (and automation/outsourcing is a tool), the idea can be abused.

I am lucky enough to work for a com­pany that is small, agile and under­stands these things. Both the ben­e­fits and the pit­falls of the­ses ideas.

Just a ran­dom thought.

One machine can do the work of fifty ordinary men. No machine can do the work of one extraordinary man. -Elbert Hubbard I find this to be very, very true when I talk to people about 100% automation goals, and outsourcing of iterative/isolated tasks. People always respond "automation/outsourcing will take my job!". My reply is always ...

Some ran­dom thoughts that have been brew­ing in my head since fly­ing in on Wednes­day around how to build out a log­i­cal series of tests, and libraries within the Qual­ity Assur­ance space. (Sorry, this is a bit of a stream of con­scious­ness post)

First, let’s cover the assump­tions you need to make:

• All tests, and test cases, will be even­tu­ally deprecated
• Every Test has some value (even dep­re­cated tests)
• Hav­ing a clear strat­egy for dep­re­cat­ing tests and then migrat­ing those tests into a bin for “func­tional regres­sion tests” is key

Start­ing here, and think­ing about it, you have to make some log­i­cal par­al­lels, the core of which is “if each test case’s value even­tu­ally reaches slightly more than 0, but future test cases must build on those test cases, mod­u­lar­ity and dis­pos­abil­ity is key”.

That’s not any­thing ground break­ing, in and of itself — how­ever, how you actu­ally imple­ment this is key. What I see Perl devel­op­ers com­monly do (and most other peo­ple) is to iso­late core behav­iors within a core series of libraries. How­ever, I reg­u­larly see lit­tle mod­u­lar­ity within those libraries.

You must approach this from the stand­point of each test case is a series of Atomic “stand alone” actions keyed into a spe­cific sequence result­ing in a test case. If this is the case — shared libraries for core actions are key. How­ever, iso­la­tion and mod­u­lar­ity of these libraries is key.

If you approach this cor­rectly, your tests become ulti­mately dis­pos­able. Say you have a test case like this:

• write 5k files
• read 4.5k files
• delete 3.5 files
• Hash remain­der files

If you iso­late the write, read, delete and hash actions into a series of mod­u­lar libraries, the actual imple­mented test case becomes a sim­ple wrap­per around these actions. Break­ing up the libraries into log­i­cally sorted libraries means that you can eas­ily swap out/around those libraries out from under­neath the tests (com­monly, a result of refactoring).

Men­tion­ing refac­tor­ing brings up an inter­est­ing side point — IMHO, QA test code, and libraries should be refac­tored con­stantly, and mer­ci­lessly. They com­monly need to cover the width and breadth of the PuT (prod­uct under test), and prod­ucts grow and change. From what I have seen, you com­monly need 2x the core prod­ucts code base to effec­tively auto­mate the test­ing of the PuT, and this means you have a huge code base (and big code bases move slow) but with 2x the code, you have to move 2x the speed of the prod­uct just to keep up with new fea­tures, dep­re­cated fea­tures, or new actions.

I should refac­tor this post later — but in essence, I am sim­ply act­ing as a cheer­leader for QA to adapt highly Agile and RAD meth­ods of devel­op­ing tests and tools. If you have >85% automa­tion (as my cur­rent com­pany does) you have to be fast to recode/tool those tests, libraries and tools, as well as devel­op­ing new ones.

You must mod­u­lar­ize, you must refac­tor, and you must dep­re­cate. Just like a devel­oper. QA is a busi­ness that has mul­ti­ple mas­ters. You must act as a devel­oper, and think like one when writ­ing test code. But you also have to leave your QA hat one.

Iron­i­cally, large code bases within QA means you should prob­a­bly think about writ­ing tests for your tests. Oh, the fun.

Some random thoughts that have been brewing in my head since flying in on Wednesday around how to build out a logical series of tests, and libraries within the Quality Assurance space. (Sorry, this is a bit of a stream of consciousness post) First, let's cover the assumptions you need to make: All tests, and test ...

Ah, the grind.

I took a few weeks off — Xmas and New Years break for me. Prob­a­bly the longest amount of time off I’ve taken off from work/coding/tech in awhile. Part of it was hol­i­days, but I think a part of it is the Burn.

We’ve all felt the burn. It’s that feel­ing you get when you open the same code base/test code/bug list for the mil­lionth time in six months, and you’re star­ing at the same chunk of code, or run­ning the same test and you have to men­tally gag on it.

I’m young, so I tend to get it in my head that I’m immune to the burn. Lately, I’ve started think­ing of releases as a Prize Fight — a box­ing match.

You start off round one — you begin feel­ing out your oppo­nent, after all — things just started and you have to gauge what’s going to be com­ing down the pipe in the next few rounds. You throw a few punches, you do your research. Things don’t really start until the end of the round.

Rounds 2–4 tend to get a bit more dirty. By now you’ve fig­ured out the easy parts of your oppo­nent, and you start throw­ing your known punches. Things start hurt­ing, you hit, he hits, you hit, he hits. You’re both hav­ing the crap beaten out of you, but you know there’s a light at the end of the tun­nel. It’s dance. Punch, jab, jab, body shot, uppercut.

Then things start to wear on. You’re tired, and you make mis­takes, you get blood­ied, but you keep going. By now, you’re not inno­vat­ing — you’re main­tain­ing. It’s impor­tant to main­tain the sta­tus quo as you progress — you have to throw the same punches over and over and over in hopes your oppo­nent falls.

Later, you fall back on instinct. By now, you’re ribs are aching, pos­si­bly bro­ken and your face looks like you got hit by a truck. But you’re run­ning on empty. You’re run­ning on pure instinct, clos­ing down the same punches you know are com­ing, and try­ing to find any gap that will win you the fight.

Then, it goes into overtime.

In the soft­ware world, this is a slip. A sched­ule slip due to new fea­tures, or late ones. It’s the fact the prod­uct doesn’t with­stand your con­tin­ued punches — the same ones you have thrown for the past months, but the prod­uct sim­ply breaks, or worse yet — breaks in new ways under the old ways.

The slip takes the most out of you. By now you just want to lay down — you want to work on some­thing else, or you want to just kick it out the door no mat­ter what. In a fight, you see des­per­a­tion — why won’t he just let his guard down for one minute so I can slip in there and knock him out?!

The slip, I think, causes the most burn. It’s the rep­e­ti­tion and con­tin­ued per­ceived fail­ure in the eyes of oth­ers that gets under your skin. Even­tu­ally, one of you will fall — gen­er­ally, it’s your oppo­nent. Whether you chop fea­tures or back off of an aggres­sive release, one of you goes down.

Fight­ers recu­per­ate. They take a break, they see a doc­tor, they mend them­selves for the next fight. Some­times they fight with frac­tured skulls, bro­ken hands or bro­ken ribs — but they’re a lull before the next fight. They keep them­selves fit and moving.

In soft­ware, espe­cially in QA, some­times — nay, most of the time — you don’t get the lull after the fight. You have to go, bro­ken and blood­ied to the next fight. Even if it’s main­tain­ing and sup­port­ing a pre­vi­ous release — devel­op­ment has moved on. QA always lags behind in a struc­tured envi­ron­ment, the job is often more inten­sive and try­ing then that of devel­op­ment, and gen­er­ally slower.

You have to move to the next fight, still tired and still hurt. You win that fight. You move onto the next, you win that one — but by now, you’re arms feel like lead and you aren’t see­ing so well.

And so on.

Don’t think I am say­ing this about my cur­rent job per-se — I’m not. Just because I’m feel­ing the burn doesn’t mean my employer burned me, or they don’t give me rest. In my per­sonal case, I’m the one dri­ving myself from fight to fight to fight.

I just think that in the case of larger, struc­tured devel­op­ment teams, this is the proper anal­ogy. Some com­pa­nies do bet­ter, some do much, much worse. But I still think it boils down to the fight.

QA peo­ple are gen­er­ally not as advanced as devel­op­ers — and fre­quently com­pa­nies treat QA as a novel idea, but not core to the busi­ness — this makes the anal­ogy above much more true, and much worse in many respects.

Luck­ily, although I am the one dri­ving myself from fight to fight, my employer cares about QA quite a bit. They ded­i­cate resources and time, but that doesn’t stop the fact we have to keep soft­ware going out the door as fast as we can make it, it’s a startup — you have to keep running.

I’m tired, and it’s late in the fight. This is the 6th of the night and I want to go home but I am not going to give up, and I am not going to stop fight­ing, fight­ing is fun. Soft­ware, tech and devel­op­ment, is fun.

Fun, inter­est and learn­ing can let us drive our­selves past our break­ing point — but there’s always a cliff sit­ting there — wait­ing for you to real­ize that you just went past the point of safety. You have to watch out for that cliff, you have to know your bound­aries and most of all, you have to know your opponent.

Edit: Let me add — one thing Box­ers, run­ners, devel­op­ers, etc all do, which I have not done to date, is to find a pace. When you ramp up, you can’t blast your­self, run­ning at full speed right out of the gate, you’ll hit the wall, hard and fast. Every­one needs to find a pace, includ­ing me.

Ah, the grind. I took a few weeks off - Xmas and New Years break for me. Probably the longest amount of time off I've taken off from work/coding/tech in awhile. Part of it was holidays, but I think a part of it is the Burn. We've all felt the burn. It's that feeling you ...

Another crazy week. I’ve actu­ally been fairly Python-Active, but instead of (again) work­ing on those things I can make public-domain, I’ve con­cen­trated on cer­tain inter­nal project.

Hope­fully, in the upcom­ing months, I can get a release to let me talk about some of the ideas and con­cepts I’m hack­ing on and I can them really start talk­ing about them.

In any case — Based off this entry: Agile Test­ing: Drucker on agility and testing

I picked up a cou­ple of Peter Drucker books from Ama­zon and I’ve started read­ing them. I pre­vi­ously had no knowl­edge of this man, but in start­ing to read this stuff, I see why peo­ple like him (and his thoughts) so much. A lot of it is amaz­ingly applic­a­ble to the space I am in at the moment.

Also — you should check out the Python411 Web­cast — I’ve started lis­ten­ing to the series on my drive, and it has a lot of good infor­ma­tion. Between this and the QA Labs pod­cast, I’m get­ting a lot of “Good Ideas” about Python/QA projects and improvements.

I man­aged to sit down and watch the vid cast for the Tur­bo­gears 20 Minute Wiki — and where I was sim­ply con­fused before (I’m not a web-programmer by any means) now I am com­pletely daunted. Tur­bo­gears looks damned nice — I just don’t know how easy it is going to be for a non-web pro­gram­mer to really get “into” it.

Another crazy week. I've actually been fairly Python-Active, but instead of (again) working on those things I can make public-domain, I've concentrated on certain internal project. Hopefully, in the upcoming months, I can get a release to let me talk about some of the ideas and concepts I'm hacking on and I can them really ...

Man. Have you ever had one of those weeks that makes you feel like your brain just got backed over by a semi-truck? That’s what this week was for me. Between seem­ing jug­gling angry bears, every­one around me seems to have come down with the creep­ing crud.

Of some inter­est, I’ve dis­cov­ered a series of QA-Related blogs (the result­ing read­ing of these makes me ques­tion my own method­olo­gies) — includ­ing a QA Pod­cast (in one episode they cover Sele­nium). Here’s that roundup:

Col­lab­o­ra­tive Soft­ware Test­ing
James Bach’s Blog
Agile Test­ing Blog
QA PodCast

Most of these con­cen­trate on the con­cept of Agile Test­ing, and basic to advanced method­ol­ogy. I know they made me think about var­i­ous approaches. Over­all, I think a lot of what’s said inside the com­mu­nity is applic­a­ble to most soft­ware development/QA groups — but I strongly dis­agree with some of the assump­tions (but I’ll cover some of that in later posts).

Good Info!

I’m hop­ing to be able to cycle back into PyHack mode next week — I haven’t been able to make progress on any of my python projects (both inter­nal and exter­nal), so my abil­ity to talk about any­thing new or inter­est­ing is greatly diminished.

One gen­eral thing I have been Navel-Gazing about is this:

My com­pany (and a few oth­ers I know about) have been look­ing for solid Python peo­ple — QA peo­ple who can pro­gram in python (for tools devel­op­ment, test automa­tion) and in gen­eral com­pe­tent “Python Hack­ers”. After scour­ing the inter­net, and post­ing every­where we can think of, it looks like a employee-market. The num­ber of skilled Python peo­ple is either incred­i­bly low, or those who are skilled, are not look­ing (or not look­ing for some­thing out­side of core development).

Over­all, it’s sort of depress­ing (espe­cially as I need the help!) but then again — I real­ize the tra­di­tional stigma of the QA role (and how do scream “it’s not like that here” loud enough) and I also real­ize that QA is con­sid­ered a step­ping stone posi­tion for many on the path to development.

Like I said — Navel Gaz­ing. I just want to find good QA Engi­neers who can also write code. I know they exist, dangit.

Man. Have you ever had one of those weeks that makes you feel like your brain just got backed over by a semi-truck? That's what this week was for me. Between seeming juggling angry bears, everyone around me seems to have come down with the creeping crud. Of some interest, I've discovered a series ...