Design By Contract Assertions Vs Unit Tests Vs Types

Tests and Assertions are similar in that Tests and Assertions are different in that (overstating a little) Assertions are also related to types (the notion of type here includes const modifier) ''(actually, Lisp rarely does this for integrity or consistency. Normally this is more to get out of a recursion down a list, or some situation where you want to tell that you're at the end of a list. Assertions to check types in Lisp are considered "poor form"). Lisp programmers don't worry too much about checking types because all lisp objects know their own types, and the environment will raise a condition if you do something which is disallowed (say (+ 2 nil)).'' Comments?

-- AamodSane, DaveHarris

I added a few. I don't think anything in UnitTests quite corresponds to pre-conditions.

"The amount of computation you can do in an assertion is limited" - I don't follow this. If you replace an assertion with a unit test, how does that give you good error handling in live code? The unit test isn't even present in live code. -- DaveHarris

A: Each UnitTest is executed only once (every time you run the tests). But assertions are run every time that piece of code is executed, even if you're not "running tests." So computationally expensive assertions in frequently used modules can really slow a program down. Also, the increase in physical program size can slow execution too, due to swapping. -- JeffGrigg

You can always mix them: let the unit test define the stimulus and the assertions define the expected results. The problem is that you can't then run the unit tests on the production build. This is solved using the DUT vs reference approach (i.e. run both versions of the code with the same stimulus and compare the outputs). Placing the assertions in the code is a valuable debug aid and does not reduce the ability to use unit tests. (AcceptanceTests must, of course, remain as an independent source of expected results). -- DaveWhipp

It will be said good unit tests are a substitute for type checking and design-by-contract. This presupposes facts not yet in evidence by saying the unit tests are good. If they are not good then the whole pyramid of assumptions based on good unit tests will collapse. My experience is UnitTests are not even as good as "normal" code and they don't get maintained by new programmers as they take over the code. It's clear some people are capable of very high quality with unit tests, but on average I bet this isn't so, which means other mechanisms are needed as well. Unit tests can't be made the answer to every issue of quality and design.

Can you describe a method that if not used or "maintained by new programmers" that will be successful? I believe every method assumes its use for it to be successful.

Compiler based type checking is not maintained by developers. DBC is not maintained by developers.

Consistent use of types is maintained by developers. I have seen too many C/C++ systems where everything is passed as an int and with random changes between signed and unsigned; or char, short, int, and long; or that do not use const. I say this not as a purist, but as one who has had to track through code trying to figure out why it doesn't work under some scenarios. I can't say for certain, but I would also guess that design by contract would also fail if it were ignored by developers.

Yes, it is easy to get into a situation where UnitTests are not used by new developers. One important principle is that it should be more difficult to not run the tests than to run them. For example, I like to have a makefile where the "build exe" step has a dependency on the "unit tests passed" rule. The flow is that the makefile first creates .o files (the compiler may catch its errors at this point), and then links these with the unit tests (and dies if it fails), and then runs the unit tests (and dies if any fail). The average developer can then treat uit-test errors in the same way as any other built-time error.

The more difficult issue is to ensure the tests are maintained as change continues. I don't think there's a technical solution to this: its a question of culture. -- DaveWhipp

I think we should think about static typing (including const checking, as I added above) and asserts as specifying the conditions that should be always true in our system. Static typing and ExtendedStaticChecking are evaluated at comile time, normal asserts are evaluated at run time. The static typing rules come with the compiler and are inflexible, asserts and ExtendedStaticChecking are custom-built. We can consider here also programs for style validation and advanced error detection like lint family.

UnitTests on the other hand verify conditions that are true in each particular context of the test and work on a different level of granularity. They cannot be used for static verification as they depend heavily on the context that is not known yet.

I believe all these categories of program verifications can be generalized in a coherent and adaptive framework. Java will incorporate asserts in JDK 1.4, and the language construct can be used also for ExtendedStaticChecking - this would be definitely a step in the right direction. Another example would be if the TestingFramework 'notices' certain things are always true during the testing, and tries to assert them speculatively. -- NikolaToshev

DaveHarris wrote: I don't think anything in UnitTests quite corresponds to pre-conditions.

MockObjects are used to test preconditions in unit tests.

Could you provide more detail? It seems to me that MockObjects can verify pre-conditions for the objects you collaborate with, but not for your objects. Verifying preconditions for a method in your code should mean checking if certain conditions are always fulfilled at invocation of this method. Correct me if I am wrong.

See DoesUnitTestingMakeStaticTypingLessUseful (which might be a good home for some of the content here (?))

View edit of April 29, 2006 or FindPage with title or text search