« June 2004 | Main | August 2004 »

July 27, 2004

5 Ways To A Slower Test Suite

One of the biggest challenges I've seen for development teams, especially those dabbling in the Agile arts, is keeping a project's test suite lightweight enough that it can be run quickly (< 10 minutes) many times a day prior to a commit. So far, I've seen no examples of really well tuned test suites of a decent size (e.g., somewhere in excess of 1000 tests). Conversely, I do have a considerable amount of experience in using and contributing to bloated, poorly-designed test suites, with execution times well over 30 minutes.

So should you wish to follow in my footsteps and develop a really bad test suite, here are my top 5 items to follow...

  1. Accept long test suites as a necessary evil
  2. Let your testing architecture emerge without refactoring
  3. Don't unit test when you can integration or system test
  4. Ignore common techniques like stubs and mocks
  5. Don't practice TDD

Accept long test suites as a necessary evil

By definition, enterprise-size applications require a lot of testing. This task will probably require a number of different testing technologies, which are needed to test a number of different implementation technologies (e.g., databases, messaging systems, web pages, etc). All in all this adds up to a lot of complex code needed to ensure the quality bar for the code base remains as high as possible.

However, none of this means the test suite should become a hideously bloated slug that takes a Friday afternoon lunchbreak to complete before anyone can commit changes to version control without fear of breaking the build.

A slow test suite should be looked upon like any other piece of poorly performing code; whenever alarm bellls start faintly chiming to suggest your test suite isn't the svelte figure it was back in Iteration 2, the suite needs to be profiled holistically to determine where the areas ripest for optimization are hiding.

Let your testing architecture emerge without refactoring

I'd originally titled this one "let your testing architecture emerge without rigourous planning", but then I realized how decidedly predictive :-( that sounded. There is no reason why normal Agile design practices cannot be applied to the test suite: hence the rename to include refactoring.

No one should be under the mistaken impression that test suties don't have architecture. In all senses of the term, they certainly do. How you layer your tests, if/where you decide to use stubs and mocks, the in-container versus out-of-container demarcation, usage of 3rd-party testing libraries... these are all architectural issues that need to be addressed as the test suite evolves.

The project technical lead needs to be across these issues and be ready to refactor and/or redesign the test suite when test suite bloat start to appear.

Don't unit test when you can integration/system/acceptance test

Integration, acceptance and system tests are greedy little buggers. They like nothing more than taking a deep, deep breath, diving head first into the presentation layer of your application and swimming all the way through the subsequent layers until they hit bedrock at the database (usually), whereupon they turn around and come back again. Sometimes they repeat this process several times in a single execution, finally informing you (a minute or so later) that your transaction was successful.

Whilst this form of testing is valuable in it's place, too much emphasis on testing in this manner is a guaranteed path to a lengthy suite execution.

Unit tests, on the other hand, are weak endurance swimmers, who are nevertheless extremely nimble over short distances. It's true that cannot cover the same wide range of aspects as multi-tier tests, but they generally execute one or two orders of magnitude faster than your typical integration test, so time spent getting a proper unit test suite in place will greatly reduce the amount of effort required for the higher level tests.

Ignore common techniques like stubs and mocks

Whether you're an in-container testing fanatic or not, you cannot deny that judicious use of both stub and mock objects can make vast differences to the execution speed of tests, not to mention enabling some form of tests that are normally difficult to achieve (e.g., exception handling tests).

So, once you've proven that an entity EJB method will return a given set of objects from the database for a given set of parameters, do we really need to make the same expensive call from the session EJB acting as it's facade?

And also from the Struts Action controlling communication with the session EJB?

And also from the JSP acting as the view for the Struts Action?

If the answer is "no". then consider speeding up the latter tests with at least a mock or stub EJB, and possibly other equivalents at the levels above (e.g., the session EJB).

Don't practice TDD

Difficult though it may be to get your head around initially, I can think of no better mechanism to generate a fast, very well-tested set of code. Good use of TDD will result in small, single function blocks of code that can be tested independently of other functionality. Additionally, TDD almost mandates the development of classes that support the use of stub and mock implementations through dependency injection.

I wont preach the TDD gospel anymore apart from point people towards Kent's book if they need any more convincing.

Conclusion

The risks associated with long-running test suites areplentiful:

Even given these issues, it's not uncommon for large projects to have test suites that take exceedingly long periods to execute. The solution: merciless refactoring. Like all refactoring, you don't need to do it all at once. Introduce a mock object here, make a concered effort to TDD the next storycard you're given, remove a redundanct acceptance test there and, little by little, the execuition time of the suite will cease to grow at such an alarming rate and may even start to recede.

Like all refactoring, test suite optimization isn't ever complete. There are always more tests coming along and always better ways to write them.

Posted by Andy Marks at 12:11 PM | Comments (1)

July 16, 2004

A trim(), a trim(), my kingdom for a trim()!

Quick rant to get the weekend started on the right footing:

In the spirit of becoming an even more service oriented company, the good souls behind the Commonwealth Bank's Netbank system have chosen to skimp on measly tasks like sanitizing data prior to validation on their transfer money page...

The end result? Whilst they have no qualms about transferring amounts such as "13", it is deemed "illegal" to request a transfer of "13 " (note the space).

As personal service to my bank (after all, they've been inconveniencing me for so long, I feel it's time for me to give something back), here's some free Java consulting:

transferAmount = (transferAmount == null) ? null : transferAmount.trim();

Posted by Andy Marks at 09:08 PM | Comments (1)

July 14, 2004

Caution! Test in Progress

Robert Watkins recently blogged about adding annotations to JUnit which I commented upon at the time, but I've just thought of my numero uno use for annotated unit tests should such extra functionality be added.

I want to be able to mark tests as work-in-progress.

Now bear with me - the reason I want to be able to do this is so I can commit these tests to CVS and not break the build because Cruise Control has been instructed to ignore any failures in tests annotated with @in-progress.

With these minor changes to JUnit and Cruise, I'd be able to follow Uncy Kent suggestion and use a broken test as a software bookmark when I leave work at night. The next morning I'd come in, look for my broken WIP test and pick up from there.

Obviously, I can (and do) use broken tests for this purpose now, but I don't like leaving work at night with any dirty laundry hanging out of CVS, so being able to commit my WIP tests and associated code (which hopefully shouldn't break any other tests, right :-)) without incurring the wrath of the resident fascist Build Master has inherent appeal.

Such a tool would also allow the acceptance tests which we hopefully all have delivered along with our shiny new story cards to be committed to source control well in advance of the code, again without breaking the build.

Posted by Andy Marks at 06:07 PM | Comments (1)

July 11, 2004

TagTestHelper - you mock me!

I have to admit I'd already decided exactly how this post was going to end well before starting the coding to support my hypothesis. Having now finished the coding, I've completely changed my mind!

I was going to make the case for not making testing infrastructure overly complex unless it really needs it. The practical context I used is testing custom JSP tags and the difficulties in correctly creating the PageContext object necessary for a complete test. My general approach to these things was:

  1. extract a method out of doStartTag to handle the processing and output required by the tag (for the sake of the argument, let's call this method generate()
  2. pass as parameters to generate() all the values it needed to complete it's operation, incuding a Writer of some sort to hold the result
  3. invoke generate() directory from the unit test, supplying a StringWriter object to hold the result
  4. Trust that generate() will be called appropriately from doStartTag() or there will be an acceptance test which will break if this is not the case.
Here's my test tag already constructed to support this method of testing.
01 public class TimestampTag extends TagSupport {
02     private static final SimpleDateFormat TIMESTAMP_FORMAT = new SimpleDateFormat("dd/MM/yyyy HH:mm");
03 
04     public int doStartTag() throws JspException {
05         try {
06             generate(Calendar.getInstance(), pageContext.getOut());
07         } catch (IOException e) {
08             throw new JspException(e);
09         }
10         return SKIP_BODY;
11     }
12 
13     public void generate(final Calendar timestamp, final Writer writer) throws IOException {
14         writer.write(TIMESTAMP_FORMAT.format(timestamp.getTime()));
15     }
16 }

I haven't had to write a tag so far that's needed testing any deeper than this and I was convinced that alternative techniques (using mocks, in particular) was going to add way too much overhead to the test to make the added advantage of being able to call doStartTag directly worth the excess baggage.

Background duly dispensed with, I've included sample tests using both my naive approach and using mock objects below.

01 public class TimestampTagTest extends TestCase {
02     private TimestampTag tag = new TimestampTag();
03     private TagTestHelper helper = new TagTestHelper(tag);
04     private static final Calendar NOW = Calendar.getInstance();
05     private static final String EXPECTED_RESULT = new SimpleDateFormat("dd/MM/yyyy HH:mm").format(NOW.getTime());
06 
07     public void testTagContentsNaive() throws IOException {
08         final StringWriter writer = new StringWriter();
09         tag.generate(NOW, writer);
10         assertEquals(EXPECTED_RESULT, writer.toString());
11     }
12 
13     public void testDoStartTagReturnsCorrectValue() throws JspException {
14         helper.assertDoStartTag(TagSupport.SKIP_BODY);
15     }
16 
17     public void testTagContentsMock() throws IOException, JspException {
18         helper.getOutWriter().setExpectedData(EXPECTED_RESULT);
19         tag.setPageContext(helper.getPageContext());
20         tag.doStartTag();
21         helper.getOutWriter().verify();
22     }
23 }

As you can see, there is very little difference between the size of the two mechanisms for testing the tag (testContentsNaive versus testContentsMock). I must admit the TagTestHelper class makes all this sort of stuff much, much easier, although I still contend the naive method results in a slightly easier-to-read test. Given this, I'm about to abandon my naive tag testing idiom and use the TagTestHelper from now on.

Note: using an alternative tag testing approach like that provided by the in-container Cactus framework can also produce tests of a similar quality to those allowed via mock objects however the effort required for configuration is substantially greater. That said, there is probably a place for some form of in-container testing in all J2EE projects.

Posted by Andy Marks at 06:08 PM | Comments (2)

I second the motion!

As Jon has mentioned previously in a blog far, far away, Jay Allen is a hero! Nuff said.

Posted by Andy Marks at 01:41 PM | Comments (0)

July 07, 2004

Scroll much?

Does anyone give two hoots about the lexical ordering of code within a source file anymore?

When I was a mere slip of a lad, only knee-high to a grasshopper, I can remember being taught that the correct sequencing of elements within a source file was vital to the rapid comprehension of the contents of the file when the next poor weenie comes along to debug your mess in future times. Of course, some ripe old languages (i.e., COBOL) positively insist your divisions are toeing the company line, but most other languages are fairly lax with such things.

The general format for the perfectly readable object-oriented source file was something along the lines of (from top to bottom):

Obviously, the above list is customized for an object-oriented language like Java, but you get the general idea.

My point is: when using tools with powerful intra- and inter-class navigation features like IntelliJ and Eclipse, source files cease to be viewed as sequentially accessed beasties and become more like randomly accessed data structures, akin to a hashtable. With the navigation techniques available in the tools above, an experienced developer is never more than a keyboard shortcut away from any element in the current source file and only one step removed from the contents of any related file.

I literally cannot remember the last time I used the scroll bar as an aid to finding something in a source file. Automated support for jumping to method declarations, finding method usages and viewing the outline of a class is, IMHO, as valuable a feature as the more frequently lauded refactoring support provided by these IDEs.

As a result of this, the layout of my class files has somewhat... degraded over time. I thank the Good Lord every day that no-one has invented a Checkstyle rule to impose the sort of ordering I mentioned above!

Posted by Andy Marks at 06:03 PM | Comments (2)