Blah.Blah.Blog

Accept long test suites as a necessary evil

By definition, enterprise-size applications require a lot of testing. This task will probably require a number of different testing technologies, which are needed to test a number of different implementation technologies (e.g., databases, messaging systems, web pages, etc). All in all this adds up to a lot of complex code needed to ensure the quality bar for the code base remains as high as possible.

However, none of this means the test suite should become a hideously bloated slug that takes a Friday afternoon lunchbreak to complete before anyone can commit changes to version control without fear of breaking the build.

A slow test suite should be looked upon like any other piece of poorly performing code; whenever alarm bellls start faintly chiming to suggest your test suite isn't the svelte figure it was back in Iteration 2, the suite needs to be profiled holistically to determine where the areas ripest for optimization are hiding.

Let your testing architecture emerge without refactoring

I'd originally titled this one "let your testing architecture emerge without rigourous planning", but then I realized how decidedly predictive :-( that sounded. There is no reason why normal Agile design practices cannot be applied to the test suite: hence the rename to include refactoring.

No one should be under the mistaken impression that test suties don't have architecture. In all senses of the term, they certainly do. How you layer your tests, if/where you decide to use stubs and mocks, the in-container versus out-of-container demarcation, usage of 3rd-party testing libraries... these are all architectural issues that need to be addressed as the test suite evolves.

The project technical lead needs to be across these issues and be ready to refactor and/or redesign the test suite when test suite bloat start to appear.

Don't unit test when you can integration/system/acceptance test

Integration, acceptance and system tests are greedy little buggers. They like nothing more than taking a deep, deep breath, diving head first into the presentation layer of your application and swimming all the way through the subsequent layers until they hit bedrock at the database (usually), whereupon they turn around and come back again. Sometimes they repeat this process several times in a single execution, finally informing you (a minute or so later) that your transaction was successful.

Whilst this form of testing is valuable in it's place, too much emphasis on testing in this manner is a guaranteed path to a lengthy suite execution.

Unit tests, on the other hand, are weak endurance swimmers, who are nevertheless extremely nimble over short distances. It's true that cannot cover the same wide range of aspects as multi-tier tests, but they generally execute one or two orders of magnitude faster than your typical integration test, so time spent getting a proper unit test suite in place will greatly reduce the amount of effort required for the higher level tests.

Ignore common techniques like stubs and mocks

Whether you're an in-container testing fanatic or not, you cannot deny that judicious use of both stub and mock objects can make vast differences to the execution speed of tests, not to mention enabling some form of tests that are normally difficult to achieve (e.g., exception handling tests).

So, once you've proven that an entity EJB method will return a given set of objects from the database for a given set of parameters, do we really need to make the same expensive call from the session EJB acting as it's facade?

And also from the Struts Action controlling communication with the session EJB?

And also from the JSP acting as the view for the Struts Action?

If the answer is "no". then consider speeding up the latter tests with at least a mock or stub EJB, and possibly other equivalents at the levels above (e.g., the session EJB).

Don't practice TDD

Difficult though it may be to get your head around initially, I can think of no better mechanism to generate a fast, very well-tested set of code. Good use of TDD will result in small, single function blocks of code that can be tested independently of other functionality. Additionally, TDD almost mandates the development of classes that support the use of stub and mock implementations through dependency injection.

I wont preach the TDD gospel anymore apart from point people towards Kent's book if they need any more convincing.

Conclusion

The risks associated with long-running test suites areplentiful:

• Developers don't run the test suite because it simply takes too long: instead they commit their changes after running a small subset of tests likely to fail... sometimes the build breaks, sometimes it doesn't. This could be deemed the Russion Roulette approach.

• Developers spend too much time running the test suite and not enough time adding functionality.

• The worst of both scenarios: developers run the entire suite, but between suite kickoff and completion, more changes have been made to the central version control repository. At this point, either the developer has the patience of a monk, or the Russian Roulette starts in earnest.

Even given these issues, it's not uncommon for large projects to have test suites that take exceedingly long periods to execute. The solution: merciless refactoring. Like all refactoring, you don't need to do it all at once. Introduce a mock object here, make a concered effort to TDD the next storycard you're given, remove a redundanct acceptance test there and, little by little, the execuition time of the suite will cease to grow at such an alarming rate and may even start to recede.

Like all refactoring, test suite optimization isn't ever complete. There are always more tests coming along and always better ways to write them.