Knowledge Center

Good Testing Starts with Good Testers – And Gets Even Better with the Right Metrics

Is there any place the 80/20 rule doesn’t apply? Perhaps a small number of clients account for a huge amount of your business. Or a few of your vendors are responsible for the overwhelming number of the headaches you face. Either way, you get the picture.

Software testing is no exception to the 80/20 rule. In fact, it’s more like 90/10, which is one thing that makes hiring software testers so difficult. Most testing work is very mundane and requires someone who has a laser-like focus, who knows how to break software, and who can document how to break it again on-demand so that developers can easily diagnose and fix the root causes. And they must do this under conditions of extreme tedium.

But even the best test engineers can be stymied by bad managers. Best in class organizations reward test engineers who find the most bugs per unit time spent testing. If you want productive QA, you should hire for attention to detail, give your testers leeway to follow their instincts, and then get the heck out of their way.

The Temperament for Testers

Too many companies staff their testing departments with people who couldn’t cut it in software development. But a bad programmer won’t necessarily make a good tester any more than a bad carpenter or roofer would necessarily make a good home inspector.

Software testing requires the right temperament. As we said, most software testing is an exercise in boredom. Here a click, there a click, everywhere a click, click. A good manual software tester must remain alert and attentive to the tiniest details so that when the software misbehaves, that misbehavior can be quickly repeated and reported.

Even the most mundane software should get tested with a fine-toothed comb, but consider software used in commercial airliners, in medical devices, in passenger cars, in traffic control devices or in other products where mistakes can maim or kill. When you get on a plane, who would you like to have tested it? Do you want it to have been a cheap tester who was easily distracted or would you rather know that it was one who persevered through the distractions until the job was done?

OK, so it’s obvious that testing requires attention to detail, but are you aware that it also requires considerable creativity? That may sound odd at first, but think of testing like an Olympic level Easter egg hunt. At this level, all the athletes will be fast. And they’ll all know an egg when they see it. But the gold medal Easter egg hunter will be the one that can best think like the people who hid the eggs. Likewise, a truly great software tester will anticipate common developer and user mistakes and will concentrate his or her effort in those areas. He or she will find creative ways of breaking your software that will otherwise frustrate your users.

Unfortunately, too many organizations don’t understand this truth about testers. These organizations see testing as something that is boring, yes. But they incorrectly believe that it is boring because it is easy. This is a terrible misconception with potentially very costly consequences.

Organizations that make this mistake hire cheap testers. Cheap testers miss a lot of bugs. Those bugs increase support costs, damage reputations, and increase costs of sales. So the “solution” typically employed is to write scripts so that tests won’t be missed. But here’s the rub: the 80/20 rule applies to these new scripts as well. If you try this approach, only 20% of your scripts will find any bugs at all, and that mystery 20% will cost you 80% of your budget to write and run. Oh, and because it’s impossible to script creativity, even those scripts will only find 80% of your bugs, meaning that 20% will still survive to infest your support lines and your reputation. Suddenly, the “cheap QA” approach isn’t sounding so good, is it?

A better approach is to hand off the software to the test team and challenge them to break it. A good test team will always report repeatable bugs. And if you really want to improve your usability and lower your support costs, you should encourage them to report even minor bugs and areas of confusion. And you should take these things seriously.

Find Your Testing Stars

So if you want the most cost-effective testers, you’ll want to measure and reward the number of bugs they find per unit time they spend testing. Yes, testing is a skill, but it’s also an art—at least among the top testers. Your best testers will follow their hunches and find problems that most testers are not creative enough to even look for – and aren’t alert enough to notice when they happen – and can’t reliably reproduce them even if they do.

Your 90th percentile testers could be finding 3 to 5 times as many bugs as your lower 90%—regardless of their experience. Do you know who they are? Here is a chart of actual data pulled from the defect tracking system of a former client. There were two “stars” on this test team (Tester #1 actually recruited Tester #2 from a former employer). But notice how the bugs found by Testers 3-5 barely add up to the bugs found by Tester #1. And you can bet that Tester #1 didn’t make 3x the salary of any of the others!


So you may want to take a look at your testing staff, what you’re asking them to do, and how you are measuring their productivity. More importantly, how is your support team spending their time? If you have a large, expensive testing department that creates volumes of documentation and misses a lot of bugs, the quality of your product is suffering while your company is wasting money.

But what if you decided to hire only the best test engineers and measure & reward bugs found per unit time? Then your testing stars will shine. Your QA and support costs will fall. Your reputation will improve. Your cost of sales will fall. And maybe you can make your company an exception to the 80/20 rule.