Blah.Blah.Blog

« May 2004 | Main | July 2004 »

June 29, 2004

Lies, damn lies and test coverage

Let me start by saying I'm a big fan of automated software development tools. It's a difficult task - most of us need all the help we can get. I use a lot of these tools. I like most of the ones I use. I seek out new ones on occasion. Version control, continuous integration, build management, code duplication checkers, 101 different flavours of testing frameworks, code generation frameworks - I love them all...

However, people can be lulled into a false sense of security with what these tools give us, sometimes to the extent of completely switching their brains to automatic as well, instead of thinking more deeply about what these tools actually give us.

The category of tools which irks me the most in this area are for examining test coverage, the two I've used so far being JCoverage and Clover.

Now don't get me wrong, I think these tools are fabulous things and serve an extremely valuable purpose, but they can give an incorrect (or at least unverifiable) picture of the quality of the test suite. For example, here is a ridiculously simple method and an accompanying test class:

public class TestCoverage {
    public static String removeExtension(final String fileName) {
        if (fileName == null) {
            return null;
        }
        return fileName.substring(0, fileName.indexOf("."));
    }
}

public class TestCoverageTest extends TestCase {
    public void testRemoveExtension() {
        assertEquals("foobar", TestCoverage.removeExtension("foobar.txt"));
    }
    public void testRemoveExtensionWithNull() {
        assertNull(TestCoverage.removeExtension(null));
    }
}

Now, it should be obvious to Blind Freddy that there's an StringIndexOutOfBoundsException just waiting to happen in the method being tested. However, from the perspective of test coverage tools, this method is 100% covered. And when the codebase is of the size and complexity of a typical commerical application, very few missed tests can be seen as easily as this one, even assuming people have the time to trawl through the codebase and test suite in pursuit of such anomalies. And herein lies the problem: test coverage tools (at least the ones I've met) only indicate the quantity of your codebase executed during your test runs, not the quality of the tests themselves. JCoverage and Clover are great at reporting the breadth of a test suite (i.e., what is being tested), but has little to say on the depth or how aspects of the suite. But unfortunately, I hear people frequently mistaking one for the other, leading to statements like:

"Our test suite rocks!!! It's got 85% coverage according to TestCoverPro (tm)(r)(c)"

Well, no - it doesn't (demonstrably) rock. It is demonstrably covered to 85%, but that says nothing of the quality of the test suite, merely the quantity. It may indeed be a high quality suite, but I don't know of any tool that can tell me that automatically. Good test coverage is a necessary, but not sufficient factor of a good test suite. The quality of the suite is a factor of the type of data injected by the tests and the order in which the data/code is executed, not the ratio of tested lines/untested lines.

I'm not trying to say these tools are misleading us, just that people tend to optimize their behaviour for what is rewarded/measured. These tools measure test coverage (i.e., quantity) - leading teams to maximize coverage at the expense of better quality tests.

Bottom line: strive for as high a test coverage figure as you can, but once you've reached that level, don't rest on your laurels. As shown above, 100% tested code can still contain bugs, and every bug found is a red flag marking a missed test.

Test coverage is obviously only part of the story here - other development practices such as test-driven development and pair programming will naturally lead to higher quality tests. Using these sorts of practices in tandem with a thoughtfully chosen set of automated tools will greatly increase the chance of an application that is tested both widely and deeply.

Posted by Andy Marks at 05:05 PM | Comments (1)

June 28, 2004

Ohmigod - what have I done?!?

OK, in my defense:

• It's deductable - and the end of the tax year cometh
  
• IMac's do basically run *nix now
  
• Although they say size isn't everything, the 20" monitor is sweeeeeet
  
• Everything I do my development on (e.g., JDK, Eclipse, IntelliJ, Tomcat, etc) is all available for OS/X
  
• I have a vision of an iPod somewhere in the near future

So although it might have been rash purchase on my behalf, I accept the inflated price and limited expansion options and are looking forward to experiencing life from the other side of the OS fence. So far, I've gotta say it's been a highly pleasurable experience:

• Everything works straight out of the box
  
• The packaging, doco and hardware itself exudes quality in the same way as a Bang and Olufsen showroom.
  
• Not having a wheely mouse takes a little getting used to...
  
• But even with all the other inherent weirdness of the OS/X interface, you never feel too far from home when you have a terminal window close by

The only major bummer so far is - no sign of a JDK 1.5 release anytime soon :-(

Posted by Andy Marks at 08:56 PM | Comments (7)

June 24, 2004

Welcome to the Jungle

In order get my head around the changes coming in J2SE 1.5 (aka Tiger), I decided to throw together the smallest program I could think of which demonstrated these new syntactic elements. What I came up with was, unsurprisingly, simplistic bordering on the useless, but it did allow me to use all seven new language elements in a little over 20 SLOC.

The seven areas I was interested in were:

1. Generics (the big one in most people's minds, and the only one currently supported by IntelliJ 4.0.3... but more on that later)
2. Autoboxing/Autounboxing
3. The foreach style of iterator
4. Static imports
5. Enums
6. Variable length parameter list
7. C-style formatted input and output

Note: Although there are other key changes in Tiger (i.e., metadata, new concurrency libraries courtesy of Doug Lea, and easier RMI stub generation), these weren't as important to me as the basic syntactic additions, which I felt I would be dealing with more regularly.

See below for a screenshot of my IntelliJ editor window showing the source. The reason for the screenshot as opposed just including the code will become apparent a little later on.

After having read quite a lot on these new language features, and now having spent the minimal amount of time implementing them in a completely trivial manner, I have these comments to make:

• I love the new for construct. This form of foreach syntax is present in many other modern scripting languages and should be quick to pickup and use for the vast majority of Java devlopers.

• Ditto for enums. I absolutely hate having to emulate C's enum construct in Java, even following the good suggestions of Joshua Bloch. I think it's great to see this functionality subsumed into the language itself.

• Personally, the static import feature would definitely be a "nice to have" rather than a mandatory requirement. Sure, it does reduce typing when referencing static elements, but I like the scoping context that reading Color.RED gives me, rather than having to peer up into the rafters to see whether the RED I'm seeing has been statically imported from java.awt.Color or is just a poorly named constant from some other package.

• Autoboxing/Autounboxing. Has caused some contentious discussions on several forums of late. I think autoboxing is basically a good thing as it results in being able to write code that works "the way that you would expect it to". By this I mean "emposwering" the JVM to automatically make the type coercion decisions us developers had been placing into our code manually up until now. Interestingly enough, I added autoboxing into my application completely by accident when I passed the variable list of arguments and automatically cast them from primitive ints to Integer objects. Previously, this sort of problem would have resulted in a compilation error and then either an API change, or some amount of hacking to convert one to t'other.

• Variable length parameter lists. Could easily have been omitted IMHO. I like the idea of such a feature, but have so infrequently ever needed the same, even when I was originally developing in C and C++.

• OTOH, I'm a big fan of the introduction of the printf/scanf functionality from C. Like varargs, this new feature is seldom (if ever) needed given the classes in the java.text package, however being able to combine formatting and input/output into a single statement is a huge bonus.

• Lastly, to generics. I had the most trouble getting my fingers around the syntax required for implementing generics. Perhaps, my lack of experience in other template-based collection languages has hurt me here, but I found it physically hard to type the code to declare my collection of Integer objects. However, I suspect this will soon pass given a little more practice. That said, I cannot say I like the result of such typing - the introduction of copious < and > symbols was very distracting when examining the code. Given I cannot remember the last time I actually had to debug a problem that was caused by lack of type safety on collections (or similar objects), my personal jury is still "out" on this new feature.

I have a couple of closing remarks that struck me as I was conducting this exercise:

1. I can distinctly remember reading that the original syntax of the Java language was "based on C, with all the unecessary stuff left out". How far it seems we have travelled since those grand statements; from bare bones language beginnings, Java has certainly matured into a massively bloated general purpose language, with all the attendent syntactic sugar. Indeed, you could easily argue that Tiger adds nothing but sugar to the Java syntax. Either way, I'm still excited about the upcoming changes and looking forward to cutting code in 1.5.

2. You can see from the screenshot, that there is going to need to be considerable rework of IDE AST parsers prior to the final release of Tiger. With the exception of the documented support of generics, IntelliJ 4.0.3 failed to recognize any of the other new features of the language. Furthermore, I discovered a quirky new bug in it's "organize imports" functionality when an import static line exists. The end result of all this was a complete mismatch between what the compiler was telling me about my program and the information reported by IntelliJ. This leads to tremendous cognitive dissonance and an enormous drop in developer efficiency. it's almost worthwhile reverting to GVim or other such arcane editors which are more readily extensible to cope with fundamental syntax changes.

   Pallada, the JDK 1.5 version of IntelliJ, is currently available as a nightly build and looks to be progressing well towards it's stated set of features. I couldn't find any sign of a release date though.

Posted by Andy Marks at 09:39 AM | Comments (0)

June 12, 2004

Name That Cat!

As a complete tangent to my usual ravings on this blog, I have an urgent identity crisis to solve. To whit, I have a kitten that needs naming in a hurry...

My girlfriend and I welcomed two kittens into our family today and, although they both came with names, I'm not 100% sure of the one belonging to "my" one, so I'm looking to my online bretheren for assistance.

Note: although, on hearing of this plan, my girlfriend (Amelia) promptly informed me that she didn't want "a bunch of f**king geeks" naming the cat, I did tell her a bunch of us might come up with a better solution than a single "f**king geek". Besides as I told her, not all of my friends are geeks :-)

Below are a couple of photos of the kitten in question. The ginger one is mine and is currently called Amy, which does suit her, although it's a little common for my liking. She doesn't respond to the name as yet, so I figure there's scope for some feline de-pol should an alternative moniker present itself.

For context, Amy's birthday is Feb 3rd, 2004 and she's very playful, loves a cuddle and purrs like a Ducati (i.e., loud and throaty). The other cat is called Coco/Rococo (belonging to Amelia) and we got both from the same place, so they're best of friends as you can see.

A further word of warning: I'm a huge fan of pop culture-related names, but Amelia has absolute veto rights and such names are generally pooh-poohed without a moment's thought.

Posted by Andy Marks at 02:10 PM | Comments (11)

June 05, 2004

Review: Hardcore Java by Robert Simmons Jr.

A quick Google search will reveal many reviews of this book Most are either negative or mixed at best. The recurring themes are:

• The title is undeserved - the content is generally not "hardcore"
• Poor copy-editing throughout, especially the now classic "error-prone" quote on the last page
• Too thin... or too expensive... or some combination of the two
• Some nuggets of value interspersed between large mounds of filler

I tend to agree with most of the above points.

Hardcore Java is the first pure Java language text I've read cover-to-cover since Ifirst learned the language. I admit I was somewhat attracted to it by the inference that I am, by reading it, a "hardcore" Java developer, although my primary motivation was the fact that it covered three relatively new areas of the language that I'd had no experience with to-date, namely:

• Dynamic Proxies
• Reference Types (i.e., weak, phantom and soft)
• Inner classes (above and beyond standard Swing event listener-type stuff)

To this end, I was satisfied with the book as it did a good job in each of these areas.

I almost made the mistake of skipping the first chapter along the lines of "it's going to be a dreary stroll through a 'Hello, World' level application again", but I'm glad I didn't. The first chapter taught me something about using initializers I'd never been aware of before, which certainly raised my expectations of the remainder of the book.

In general, I have to say that I got something from every chapter. Sure, as most reviewers have suggested, the level of content is uneven, but for my current level of knowledge, I found the book to be a helpful resource. YMMV.

I found the length perfect as I shy away from "thick" books that adopt a "quantity over quality" approach and I basically don't have time to do them justice. I'm a huge fan of brevity and clarity in these things and would much rather a concise 300 page text than a bloated 500+ page equivalent.

Even given the succinct length, several of the sections could have been ommitted or considerably reduced in size. The final section on the upcoming (at time of publication) JDK1.5 was just being greedy and, although detailed, should not have been included given the publishing date and books dedicated to Tiger soon to be published.

Similarly, the section on Constant Objects is interesting, but with the fine work Joshua Bloch has already put into this area, and it's inclusion in JDK1.5, it too could have been left on the cutting room floor. The generally though-provoking section on the appropriate use of the final keyword is also screaming out for a rigourous copy-editing.

As for the title, the negative comments are probably warranted, although a quick flick through the chapter sections should give an astute potential buyer a good idea of the target audience. I'll have to get around to reading Effective Java (Joshua Bloch), as it seems to address a similar level of developers but with a more consistent and higher quality end result.

After finishing the book, it's my belief that it's been the recipient of some fairly harsh criticism, some of it warranted, some not. It's very easy to adopt a harder core than thou perspective when reviewing books that purport (through title or otherwise) to be of higher quality than the reams of other texts out there.

By no means is this book perfect - it reeks of a rushed publication, however you could distil 100 good pages of intermediate-to-advanced Java knowledge out of it. Given the plethora of obese Java texts in the market at the moment, that's a signal-to-noise ration I'm happy to live with.

Posted by Andy Marks at 08:23 AM | Comments (2)