Bickering about unit testing

Doubts on the effectiveness of unit testing

Unit testing has entered the programming mainstream with XUnit packages and derivations of them. They are available for all mainstream programming languages. It is not normal today shipping an OSS project without any tests. Programmers can read test cases like behavioral specifications of APIs and they often learn a lot about a system from this sort of code reading ( at least I do ).

Still unit testing is disputed as a reasonable practice by many respected programmers and I wonder if guys like Joel Spolsky or James Coplien aren’t basically right? Isn’t it true that UTs have to be permanently adapted as our code base changes and doesn’t this imply a significant maintenance overhead even and foremost in early phases? Coplien suggests design-by-contract as a more lightweight and DRY alternative to writing UTs: place pre and post-conditions directly into the code and check the available units i.e. the interface specifications. Isn’t this far more agile and won’t better coding practices make UTs go away just like many of the once celebrated design pattern go away when using powerful language level concepts like multimethods and higher order functions?

Black box testing

When you work as a tester in the industry you essentially specify and implement test-suites according to specifications. Your product is not the system under test ( SUT ). You are not interested in the inner working of a system and its components. The SUT is a black-box and the SUT code might change arbitrarily. If any code is exposed it is SUT API code being accessible by clients application like your test app. The API might even be fully away though and instead you’ll test in- and outgoing commands sent for and back between your test app and the SUT according to a specified command protocol. All of those tests are functional- or system level tests and the tested units remain hidden. As a tester you don’t care about the way the system is built but only how it behaves.

Can we use our standard UT frameworks to implement black box tests? Well, isn’t this actually their most frequent use?

Are there any UTs around?

What if the most common unit tests we are finding in the wild are functional or system blackbox tests applied to API level functions/classes, implemented in one of the available unit testing frameworks? Some of the system components are abstracted away and get replaced by mock objects representing networks or C/S databases but this just avoids system integration tests. A close reading of unit testing might indeed lead to Jim Copliens conclusion that they are better implemented as pre and post-conditions but you won’t test a system on such a fine grained level. Using UT frameworks for functional tests has short comings but it doesn’t mean they are not used for them. When the interface is kept small the likelihood that it gets badly broken when you evolve your system is manageable. This is the prime reason why programmers do not suffer from writing UTs and maintenance costs are kept under control. Every software tester in the industry knows that writing tests takes much effort and is very costly but changes in public APIs isn’t a major reason.

UTs and beyond

The missing link between between current UT systems and a test-system for all kinds of SUTs is a dataflow connection which triggers tests in a particular order. By this I mean that each test can produce data as a side-effect which can be required within another setup of a test-case. In Junit4 we have `@before` and `@after` annotations for running setups and tear-downs unconditionally. When adding two more annotations `@require` and `@provide` it becomes possible to specify conditions on running tests by means of the need of data. A test-runner has to match the @required data against the @provided ones and determines a schedule.

In case of Java this can be checked at compile time using an annotation processor. In .NET one might apply those checks once the assemblies are loaded during initialization of the test-runner. The only disadvantage of load-time checks is that all available test-modules have to be loaded initially and not on demand.