Part 9 of Bill Venners’ interview with Dave and Andy talks about how to effectively use assertions in your code. The discussion reminded me of some thoughts that were bouncing around my head a few weeks ago about the relationship between assertions and tests.

(None of this is new or original; in fact, I’m sure I’ve read all of these thoughts somewhere before. But I think they’re worth repeating.)

In late March I was finally able to sit in on Mike Clark’s talk on test-driven development. It was a great talk, but there was one question from the audience that really bothered me, because I knew there was a flawed assumption in the question, and I felt that the answer should be obvious, but neither Mike nor I could see it at that moment.

The question was from a lady who apparently had a background in the Eiffel language. She talked about Eiffel’s design-by-contract constructs: assertions for preconditions, postconditions, and class invariants. And then: “Isn’t that a better approach, so that you would have the tests actually in the code?”

It’s an excellent question. And the answer (which became crystal clear to me 10 minutes after the talk was over) is simple: assertions are not tests.

So what are tests?

Tests involve two parts: behavior checks, and input data. Assertions can partially do the behavior checking, but they don’t supply the input data. And there are very good reasons for having both parts in one place.

From a unit-testing point of view, the assertions that matter most are the postconditions. They have the primary job of verifying that the methods they are attached to did the right thing. (Preconditions can be used for correctly handling invalid input, but there are some good reasons not to use them for that).

Postconditions must be generalized: they must work for all possible inputs. In other words, the postcondition is a different (ideally more declarative) way of expressing the result of the same computation performed by the body of the method. Therefore, for a method that does fairly complicated things with its input, the postconditions must either:

  1. depend on the same helper methods as the body of the method;
  2. be just as complicated (and likely to contain bugs) as the body of the method; or
  3. be just a sanity check rather than a full validation.

And in fact, the last is most common. Consider a method, byte[] md5(String text), that calculates an MD5 secure checksum on its input. That’s a complicated mathematical operation. So consider how you’d write the postcondition. Option 1 might be practical, but isn’t much help from a unit-testing perspective. Option 2 is not really practical. More than likely, you’ll fall back to option 3, just checking that the result is 16 bytes long (because all MD5 checksums are 128 bits long) or something similar.

Unit tests, on the other hand, just have to check particular inputs and outputs for something like this. So you can supply canned input and test the results against precalculated checksums. You might calculate checksums for the test data using different tools, so that your test is more for interoperability than correctness (you’re trusting those other implementations to be correct). It might be practical to hand-calculate an MD5 checksum for a small input to round out the test.

So what are assertions?

So what are assertions, then? What are they good for?

Assertions serve three primary purposes:

  1. They serve as extremely basic tests, catching basic error conditions during development.
  2. They are internal documentation, helping readers to see what the developer understood about the code as it was being written.
  3. They are a fail-fast mechanism, ensuring that errors are reported close to where they occur.

Those things can still be very valuable, but in my opinion the least valuable role of assertions is the testing role. Assertions cannot and should not be viewed as comprehensive tests.