One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accuracy. At best you can declare a baseline, eg that your algo should be say 90% accurate (if you implemented it properly based on literature results) and if you don't get that, then error. In that case you can use a numerical metric, but other applications you might care about the result being aesthetically pleasing (eg a video ISP where you do colour correction on the stream coming from a low level camera).<p>Or hardware where the advice is usually to mock the device under test. But if you don't own the hardware the most you can do is try and emulate it, and maybe check that your simulated state machine works. In my experience its easier to run with hardware connected and just skip those tests otherwise. There are also extremely subtle bugs that can crop up with hardware interfaces like needing to insert delays into code (eg when sending serial) that will otherwise fail in the real world.<p>OpenCV has some interesting approaches to this, for example testing storing a video in a certain format, inserting a frame with a known shape (like a circle), then reading back the video and checking that the shape can be detected.
I think this is a fairly good summary of the most important testing styles and where / when to (not) use them.<p>One more category of tests I would add are meta tests (like mutation tests). These are tests which test the tests, seeing if they would actually catch any errors / bugs or just report everything to be alright always.
"If something is difficult to unit test, refactor until it's easy."<p>This is often a good idea but if you only need the flexibility to enable unit testing then it may make your system more complex than it needs to be. Only introduce indirection where it's really needed. See also "test induced design damage" and "write tests, not too many, mostly integration".
When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are:<p>1. The argument x must not be 0.<p>2. The variable x must smaller than the variable y.<p>3. The list foo must be non-empty.<p>4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call.<p>These 'invariants', or assertions, can be extremely useful for testing the correctness of the code. Put simply, if an invariant is violated (during unit test, integration tests, or system tests), it indicates that either the design or the implementation is wrong. An article on testing methodology would be more appealing if it had some discussion on exploiting invariants/assertions.
Very nice overview. I very much agree that all these techniques are very useful. But one thing still bothers me. How does one test with a notion of time? How do you test a cron or a calendar with alarm function?
I’m a complete newbie when it comes to writing tests. But I know that SQLite is tested to hell and back, and I believe Richard Hipp (the creator) has said he spends more time and lines of code on the testing suite, than on the SQLite code itself. I hope he shares some of his insights some day.