Property-based testing is great but people should approach this with equivalence class partitioning [1] first. That always seems simpler before people think about it.<p>And they should be prepared to deal with combinatoric explosion [2] with pair-wise testing [3].<p>And it's no replacement for high-level behavioral tests. Because we care most about how the software actually works for the customer.<p>[1] <a href="https://en.wikipedia.org/wiki/Equivalence_partitioning" rel="nofollow">https://en.wikipedia.org/wiki/Equivalence_partitioning</a><p>[2] <a href="https://en.wikipedia.org/wiki/Combinatorial_explosion" rel="nofollow">https://en.wikipedia.org/wiki/Combinatorial_explosion</a><p>[3] <a href="https://en.wikipedia.org/wiki/All-pairs_testing" rel="nofollow">https://en.wikipedia.org/wiki/All-pairs_testing</a>
We have a fair bit of property tests in KAZOO now, using PropEr[0], an open source implementation in Erlang.<p>Fred (of Learn You Some Erlang) wrote a great book/site[1] on property testing in Erlang and Elixir, well worth the price of the book.<p>I find property testing is a muscle; the more time I can spend writing them, the "better" I get at writing useful tests. But the muscle atrophies quickly for me, so having the existing tests helps speed up getting into the mindset again.<p>I think the model checking (proper_statem in the PropEr code) is the real winner though. We can model a part of our system, write state transitions and then test those against both the model and a running system and compare our results. Any discrepancy points to either the model or the system being wrong (which is its own joy/pain to figure out).<p>[0] <a href="https://github.com/proper-testing/proper" rel="nofollow">https://github.com/proper-testing/proper</a><p>[1] <a href="https://propertesting.com/" rel="nofollow">https://propertesting.com/</a>
Property-based testing is awesome. I try to write prop-tests by default, with unit tests for explicit 'must test'.<p>The author doesn't mention that Hypothesis directly supports combining explicit testcases with your generated testcases. So you can write something like:<p>'Test list == reverse(reverse(list))`<p>And ensure that it always tests cases like 'empty list'.(I don't recall the exact syntax and wasn't able to find it with a quick search).<p>I think this is a nice way of testing because I get to define the testcases I can think of, the edge cases I expect, and also use generated tests where possible.<p>The downside to using this approach is performance. Especially if you're using this for something that performs IO, this can slow your tests down by 100x (since you run the tests 100s of times more).
Property-based testing is amazing. However: it's rather difficult to wrap your head around it and start writing years for real-life non-toy examples.<p>Workshops and training helps, but there's not that many of those. Having a colleague who's already grokked it also helps immensely.
In praise of being paid to write unnecessary tests that break within hours after writing them and then spending hours incrementally fixing these tests as they break with every. fucking. commit