Blah.Blah.Blog

The category of tools which irks me the most in this area are for examining test coverage, the two I've used so far being JCoverage and Clover.

Now don't get me wrong, I think these tools are fabulous things and serve an extremely valuable purpose, but they can give an incorrect (or at least unverifiable) picture of the quality of the test suite. For example, here is a ridiculously simple method and an accompanying test class:

public class TestCoverage {
    public static String removeExtension(final String fileName) {
        if (fileName == null) {
            return null;
        }
        return fileName.substring(0, fileName.indexOf("."));
    }
}

public class TestCoverageTest extends TestCase {
    public void testRemoveExtension() {
        assertEquals("foobar", TestCoverage.removeExtension("foobar.txt"));
    }
    public void testRemoveExtensionWithNull() {
        assertNull(TestCoverage.removeExtension(null));
    }
}

Now, it should be obvious to Blind Freddy that there's an StringIndexOutOfBoundsException just waiting to happen in the method being tested. However, from the perspective of test coverage tools, this method is 100% covered. And when the codebase is of the size and complexity of a typical commerical application, very few missed tests can be seen as easily as this one, even assuming people have the time to trawl through the codebase and test suite in pursuit of such anomalies. And herein lies the problem: test coverage tools (at least the ones I've met) only indicate the quantity of your codebase executed during your test runs, not the quality of the tests themselves. JCoverage and Clover are great at reporting the breadth of a test suite (i.e., what is being tested), but has little to say on the depth or how aspects of the suite. But unfortunately, I hear people frequently mistaking one for the other, leading to statements like:

"Our test suite rocks!!! It's got 85% coverage according to TestCoverPro (tm)(r)(c)"

Well, no - it doesn't (demonstrably) rock. It is demonstrably covered to 85%, but that says nothing of the quality of the test suite, merely the quantity. It may indeed be a high quality suite, but I don't know of any tool that can tell me that automatically. Good test coverage is a necessary, but not sufficient factor of a good test suite. The quality of the suite is a factor of the type of data injected by the tests and the order in which the data/code is executed, not the ratio of tested lines/untested lines.

I'm not trying to say these tools are misleading us, just that people tend to optimize their behaviour for what is rewarded/measured. These tools measure test coverage (i.e., quantity) - leading teams to maximize coverage at the expense of better quality tests.

Bottom line: strive for as high a test coverage figure as you can, but once you've reached that level, don't rest on your laurels. As shown above, 100% tested code can still contain bugs, and every bug found is a red flag marking a missed test.

Test coverage is obviously only part of the story here - other development practices such as test-driven development and pair programming will naturally lead to higher quality tests. Using these sorts of practices in tandem with a thoughtfully chosen set of automated tools will greatly increase the chance of an application that is tested both widely and deeply.