« September 2004 | Main | November 2004 »
October 26, 2004
Test Suite Smells
Not running the risk of being accused of thinking of an original idea for a blog entry, this is yet another set of Agile-scented "smells" - inspired by a blog written by Jon on Development Environment Smells... welcome to Test Suite Smells. Believe me, I did try for a pun around suite/sweet, but couldn't crack it :-(
Spoiler Alert! A lot of what follows are motherhood statements - don't say I didn't warn you?
SMELL #1: Suite takes too long to run
"Well, DUH!", I hear you say. And you're right. Everyone knows a long running suite is a smell. However, the jury is still out on how long is too long. Personally, my limit would be around 10 minutes or so, anything longer and developers will start not running the whole suite, usually with SMELL #2-style consequences.
Usually caused by an abundance of long running tests. These tests are more often than not integration/system tests and may not even be suitable for the test suite. I'm not idealistic enough to think the suite should contain nothing but "pure" unit tests, but the team should aim to keep the unit test:integration test ratio fairly high (10:1?, 50:1?).
SMELL #2: CI environment is broken more often than not
This is most often a symptom of a long running test suite. If it takes some large amount of time to run the suite, the developers will start to be selective about when/how often they run the whole suite and "roll the dice" the rest of the time by just running a handful of tests around the particular pieces of code they're currently working with. Depending on how well the tests have been developed, this approach may/may not be successful. When it's not successful, it's usually related to SMELL #3.
Of course, it could also just mean you have a lazy, undisciplined pack of rabid developer hounds...
SMELL #3: 1 code logic error leads to >1 test class breakage
If you're truly testing everything in one place and one place only, then a single logic error introduced into your code should only result in tests breaking in one part of your suite. If you find yourself in the position where such a breakage ripples through the test suite like the aftershocks of an earthquake (breaking tests all along the way), then your test suite contains a large amount of duplicate testing and not enough usage of mocks/stubs to remove dependencies on objects tested in other places.
SMELL #4: Suite cannot run without a network cable
Most enterprise-scale systems require connections to external applications for various purposes; payment gateways, legacy systems, authentication servers, etc. If you're test suite requires actual live connections to these systems to run, then you've definitely crossed over the boudary into system testing. Such testing, whilst invaluable, should be moved away from the test suite.
A suite could be run in a considerably more naked mode than just a standalone workstation. You could craft a test suite that requires nothing more than the JVM, JUnit, the code base and test suite... no J2EE container, no RDMBS, no 3rd-party libraries - mocks/stubs all around. This puritan approach would certainly lead to a suite which ran like grease lightning, but risk entering the realms of SMELL #5.
SMELL #5: Over reliance on mock testing
Much like those interesting folk who prefer to immerse themselves in all forms of electronic communication to the complete detriment of any human communication, if you bind your test suite completely into a cocoon of carefully hand-crafted mock objects and interface stubs, you run the risk of forgetting what reality is really like.
The underlying assumption behind the usage of mock/stub to stand in for real implementations is that the standins do perform in a fashion largely identical to the real thing. If this is not the case, or if you don't have enough tests using the real implementation, then reality will tend to remind you of this mistmatch at the most inconvenient times and in the most embarassing of circumstances.
SMELL #6: Under reliance on mock testing
Obviously the flip side of the previous smell, if you insist on real versions of every dependency your application communicates with, then you're asking for a world of hurt in the form of a bloated, sluggish, brittle test suite.
After all, if you have one test proving that your RDBMS can return dataset A for query B, then cannot we assume it will do the same thing in each of the 10 other tests in various parts of your suite that present it with the same query?
Obviously you walk a fine line with the decision on how much reliance to place on techniques like mock testing; too much and you run the risk of missing the strength of feedback from a reality-based environment, too little and the test suite quickly becomes unmanageable. By constantly listening to the creaks and groans of the test suite and bitches and moans of the developers, you should be able to get a feeling for where the pain points are in relation to the test suite. Once thusly informed, prioritize and triage!
Conclusion
If you move in an Agile world, you put as much effort into writing test code as you do with production-destined code. In fact, prior to deployment into whatever production environment is appropriate for your codebase, the development process will have been exercised the test suite considerably more often than the codebase itself. Therefore, having the knowledge and experience to evolve a suitable test suite architecture is vital to the ongoing agility of the project team.
IMO, it is unfortunate that relatively little time has been spent concentrating on this level of software design. Reams and gigabytes have been written on production code architecture, but scant regard to the test code equivalent. As with most things in software development, good teams will come up with good solutions to these issues, but I suspect many a project team working on it's first Agile project in earnest is slowly drowning under the weight of a poorly-design test suite - a situation that could be greatly assisted by the provision of some Patterns of Enterprise Test Suite Architecture-style scribblings.
Posted by Andy Marks at 11:48 AM | Comments (0)
October 24, 2004
IoC/DI for the Time Poor
I recently gave a couple of lectures at my old university on, amongst other things, using a combination of Mock Objects and IoC/DI to enable testing of "those hard to reach places". Having received some very positive feedback on the lectures, I've created a short tutorial for anyone wanting to see how this process works.
The intended audience are Java and JUnit literate but with little knowledge of IoC /DI concepts apart from perhaps a theoretical understanding.
Enjoy.
Posted by Andy Marks at 11:54 AM | Comments (1)
October 21, 2004
I'm probably the last one to know this, but...
A learned collegue of mine on my current project has just clued me in on another little IntelliJ gem of which I'd been previously unaware:
The Ctrl-N Open Class dialog expands camel-cased abbreviations into a matching set of class names. For example, if you have a class in your project called SomeReallyCoolFunkyClass, just Ctrl-N then type SRCFC and IntelliJ will expand that to the actual class name for selection.
Should you also have a StupidRancidCrappyFloppyClass, both classes will appear in the class selection drop list.
What will they think of next!?!
Posted by Andy Marks at 07:14 AM | Comments (1)
October 18, 2004
Reducto Ad Absurdum
Let's imagine you've just brought that Lamborghini Diablo you've been lusting over for the last 10 years or so. It's one of the highest performing cars available to people with the dosh to pay for it. Given it's bloodline; big beefy sports car, you naturally expect it to do big-beefy sports car type things bigger and beefier and sportier than pretty much anything else on the market. After all, that's why you paid so much for it - performance!
So, given all this, would you then lock your pride and joy in a squallid, dank garage? Would you regularly forget to service it? After all, you're paying a small fortune for the best the market has to offer - shouldn't it therefore be able to take anything you can throw at it with aplomb and still perform the pants off anything else around?
Does the fact that it's a Lamborghini excuse the owner from paying attention to those mundane yet necessary matters of it's upkeep to ensure it's still behaving like a Diablo next month rather than a Edsel?
If you agree with that last statement, I dare say your local Lamborghini dealership (or equivalent) probably sends you a bottle of Hermitage Grange every year on your birthday to ensure your patronage the next time you're in the market after yet another knackered Diablo has been towed away from your premises.
(and here comes the tenuous connection with the general topic of my blog...)
Assuming you see the breakdown in logic in the above paragraphs, then "riddle me this, Batman": why then do generally sane IT Managers in companies the world over treat their hugely expensive employees like the personnel equivalents of the Lamborghini in my hypothetical situation above? Why is it that companies pay good money to obtain IT resources (either as direct employees or consultants, doesn't really matter) and then assume their purchase cost ensures they're capable of producing consistently quality work in any number of stupendously poor work environments?
Be it the level of ambient light, noise, desk layout, airconditioning (or lack thereof), pay, corporate policy, dress code, bureaucracy or any one of umpteen other infinitely-controllable elements, I see workplace after workplace where the employer has made huge investments into the hire of their staff and then demonstrated a staggering lack of interest/knowledge in the simple things that can ensure their highly-paid little princesses are given the best chance to perform day-in, day-out.
P.S. In no way do I consider myself a Lamborghini Diablo of developers! I'd be more than happy being the developer-equivalent of my Honda VTR1000, but even then I suspect my sites have been set too high.
Posted by Andy Marks at 08:04 PM | Comments (0)
Death by Honesty?
Every now and again, I hear stories about my fellow IT consultants suffering badly on projects as a result of merely "being honest". Some of the stories end in the people being forced off the project as a direct result of their honesty!
My initial reactions to this stories are part shock, part dismay. After all, what could possibly be wrong with being honest? IT consultancy as an industry has been tarred far too often by various practitioners acting in a less than honest manner, so it seems absurd to punish those members that are naturally honest, doesn't it?
Digging deeper in most of these cases reveals a slightly more subtle definition of "honesty" is usually at the root of the victim's woes. I dare say most of these issues are caused by someone voiceing their honest opinion, rather than a pure fact. The difference between these two is clear to most of us, but obviously a little slippery for some to grasp.
"What's wrong with even trading honest opinion?", I hear you ask? Well, my take on this is that in the consulting world, we walk a very fine line between being open and "honest" in our dealings with the customer on one hand, whilst at the same time being cogniscent of what information can/should be kept within the team.
For example, slippage against estimates is something that the customer should definitely know about, so by all means be honest with them ASAP with these matters. It might sting a little, but it's far better they find out sooner rather than later.
Conversely, disputes between members of the technical team are probably not something you would want/need to escalate unless the problems are affecting the performance of the team. I don't believe we are doing the customer any injustice by not telling them that the team is in two minds over some aspect of the project architecture. Furthermore, this sort of news might spook an already jittery customer, whereas the development team just sees it as par for the course with typically prima donna-esque developers.
Are we, as consultants, being duplicituous by not revealing these sort of issues to our customers? Some may say so (usually under the banner of "honesty trumps everything"), but I'd counter by saying that conscientous filtering of information is part of our responsibility as professionals. It is as unprofessional for us to use honesty as a lever for airing all our woes as it is for us to use the "few quiet drinks" we may have had on Sunday night as an excuse for breaking the build on Monday morning. I think being able to separate the items into the appropriate bucket is one of the things that makes a competent consultant.
As with all such issues of this nature, these grey areas are prone to abuse in the wrong hands.
The other thing about an honest opinion is that it may very well not be based on a wholistic view of the issues. An incomplete picture of things combined with an "urge to purge" has probably been the cause of many of the "death by honesty" stories I heard through collegues.
All this is just IMHO of course :-)
Posted by Andy Marks at 04:13 PM | Comments (0)
October 08, 2004
ServletTestHelper for the Time Poor
I've recently been bashing my head against the bug bear of most open source projects: lack of suitable documentation. The culprit was the MockObject class, ServletTestHelper. Checking the Mock Object home page didn't reveal anything of substance, ditto for a Google search.
I'd been itching to use ServletTestHelper since having considerable fun with it's little brother, TagTestHelper, and eventually managed to bend it to support my needs, although it took debugging into the source code in some instances.
In an effort to save some other poor sod needlessly expending the same amount of energy and time, I've created a brief article detailing the main use cases for ServletTestHelper. I haven't to covered Mock Objects or general testing design philosophies (by definition, the time poor have no time for such things) and tried to cut to the chase to get people producing tests as quickly as possible.
I hope people find some value in this.
Posted by Andy Marks at 04:32 PM | Comments (0)