jessenoller.com

As a long-time automation-engineer/test-focused guy I’ve pon­dered the great exis­ten­tial ques­tion of “how much test­ing” is enough for awhile.

More recently, I’ve started focus­ing on the cost of not test­ing a product.

Take for exam­ple, Fig­ure 1:

Let’s take a sec­ond for terminology:

• (A) Unit tests: These are tests focused on devel­oper and main­tainer pro­duc­tiv­ity. These are “close to the code” tests that run in mostly sim­u­lated envi­ron­ments. Unit tests are a cor­ner­stone of Agile method­ol­ogy — gen­er­ally speak­ing, you make these before your code.
• (B) Smoke/Simulation: These are the “next layer up” — they use par­tial sys­tems (e.g. your code + the guy’s next to you mod­ule) to run more integration-style test­ing. Smokes are nor­mally run on every com­pi­la­tion of the prod­uct along with unit tests. They do not require a fully deployed, func­tion­ing sys­tem — only a small group of parts.
• © Acceptance/Functional/Regression:
  • Accep­tance Test: These nor­mally com­prise a large num­ber of your tests
    in an orga­ni­za­tion. Accep­tance tests prove that the spe­cific
    component/feature is sane in the con­text of the fully deployed prod­uct
    – you might require these to be fully devel­oped, exe­cuted and pass­ing
    before a spe­cific com­po­nent or fea­ture is merged to trunk. Accep­tance
    tests prove that the feature/component works as intended (not
    pro­grammed). They should be short in exe­cu­tion time.

  • Func­tional Tests: Func­tional tests are “larger” and should test as
    much of the func­tion­al­ity of the feature/component as pos­si­ble, they
    should also test with an eye towards other parts of the prod­uct and
    sys­tem (e.g. inte­gra­tion). Func­tional tests should be as expan­sive and
    detailed as pos­si­ble. These can also be called Regres­sion tests.
• (D) Stress/Scalability Tests: This should be self-evident. Stress tests
  build on func­tional areas to push the prod­uct to it’s lim­its — how
  many files can it hold, how many con­nec­tions can it with­stand, etc.

• (E) Per­for­mance Tests: Char­ac­ter­i­za­tion of key per­for­mance stats:
  Objects/second records parsed/sec, and so on.

Now, I want to point out: These def­i­n­i­tions are part-agile and part-continuous inte­gra­tion. They don’t wholly mesh with ter­mi­nol­ogy used your work­place, or agile. I also know def­i­n­i­tions are a holy war, but the def­i­n­i­tions are sec­ondary to what I want to talk about. I also excluded specif­i­cally call­ing out exploratory testing.

What the hell *am* I talk­ing about?

If you look at fig­ure A, You’ll note I put “Test” (test engi­neer­ing) off to the side to rep­re­sent their par­tic­u­lar own­er­ship in this model. Unit Tests (and by most mea­sures, smoke and sim­u­la­tion tests) are under the own­er­ship of the core developers.

The other test areas are the own­er­ship of test engi­neer­ing — obvi­ously they would not exclude Dev from help­ing though (after all, they win as a team, and fail as a team) but Test is focused on ver­i­fi­ca­tion that the prod­uct is as tested-as-possible before it gets into stage F — the hands of the user.

Ok, this is all fine and good — but hear me out.

This dia­gram is about cost — for each layer the code/feature passes through ema­nat­ing from the devel­oper, the cost to the team, and the dif­fi­culty in iden­ti­fi­ca­tion and res­o­lu­tion climbs.

This is why Devel­op­ers write a lot of unit tests and check them in so they run with every check in. Right? You’re doing that, right?! The cost for a devel­oper to find a bug with a unit test, and the cost to fix that bug intro­duced through new code/refactoring/etc, is essen­tially 1.

Here’s a new dia­gram with some straw-man costs:

Essen­tially, it is in your best inter­est, as a devel­oper, as a team, to encour­age lots and lots of tests lower in the stacks shown here. It starts with com­pre­hen­sive, checked in unit tests. It con­tin­ues with hav­ing a strong, repeat­able test­ing dis­ci­pline (for which I rec­om­mend test automation).

Why? Because — as you move higher in the stack, that damned bug some­one checked in is hid­den behind layer upon layer of code. The fur­ther from the unit level a bug gets, the more com­po­nents and envi­ron­ment vari­ables get involved. The more of these that get involved, the harder it is to iden­tify and fix, and the higher the cost.

Now, your bug (our bug) has not only wasted your time, it’s hold­ing up a release, test engi­neers time (albeit — this is our job) is wasted. The higher in the stack a bug gets — the higher the cost in wasted man, release and test hours.

For exam­ple — your typo in some mes­sag­ing code man­ages to sneak its way through to the (E) Per­for­mance level. Let’s say your per­for­mance tests take, oh, a week to run to com­ple­tion. For some rea­son, this sneaky beast only pops up when your system’s clocks resync after 6 days of runtime.

So, 6 days into a 7 day test — ding fries are done — the entire sys­tem poops itself. You now have to triage the crash, you have to fix it after you iden­tify it (which is prob­a­bly going to be hard — given it’s a per­for­mance test, you shut off non essen­tial log­ging) and then you need to re run the test.

You lost 6 days. More than likely, those are 6 days of lost time you didn’t allo­cate for when you promised the fruits of this iteration/release to those wealthy swedish bankers, eh?

God help you if your bug gets to level (F). This is called the “aver­sion level” because after a few of these sneak out, and the CEO of the com­pany starts get­ting phone calls at 4am from those swedish bankers — you’re either going to get a stern talk­ing to, or some time in “the box” (all CEOs have a pun­ish­ment box).

Your goal is to avert bugs from reach­ing Level F. F stands for F’ed in the lit­eral sense.

My point isn’t just about cost. Given this tiered approach, and the need to find as many bugs as pos­si­ble, you’re going to end up hav­ing some amount of code dupli­ca­tion between the higher lev­els of test­ing and the unit/smoke level — after all, most of the tests above that level are external-system level tests.

Some code — or logic — dupli­ca­tion on a higher level isn’t always bad, given the con­text of where the code is run­ning. Not to men­tion, fre­quently, the code within the prod­uct may not be in the same lan­guage as the code that’s automat­ing the tests. Dupli­ca­tion of unit test logic on a system-test-level is always going to happen.

Yes, you can and should reuse code as much as pos­si­ble, but you can also do this through grey-box test­ing approaches (e.g. expos­ing APIs into sys­tem inter­nals you would not nor­mally have access to).

Also — this means you have to give your teams time to test. You need to give them ample time to auto­mate what is rea­son­able, and you need to be will­ing to not ship a com­po­nent or fea­ture that sim­ply isn’t ready. Much less one that hasn’t been tested.

The last thing you want is to have a bug — no mat­ter what it is — hit level F. You, our job on a soft­ware engi­neer­ing team is to put out the absolute best prod­uct pos­si­ble — and you can’t do that with­out fill­ing in all of the mag­i­cal test­ing boxes. You need to under­stand that for every step away from the code you get, the higher the cost.

Let­ting pre­ventable bugs get in the hands of users is not avoid­able — but the risk can be mit­i­gated, and many bugs that do end up in the hands of users are avoid­able. The more (and sooner) you test, the lest wealth you expend, and the hap­pier you will be. And the more prof­its you will reap. We like money.

As a long-time automation-engineer/test-focused guy I’ve pon­dered the great exis­ten­tial ques­tion of “how much test­ing” is enough for awhile. More recently, I’ve started focus­ing on the cost of not test­ing...