Property-Based Testing for the People

This work was discussed by the author in the Haskell Interlude podcast as well [0]. Highly recommended and probably easier to digest than a whole dissertation.<p>[0]: <a href="https:&#x2F;&#x2F;haskell.foundation&#x2F;podcast&#x2F;59&#x2F;" rel="nofollow">https:&#x2F;&#x2F;haskell.foundation&#x2F;podcast&#x2F;59&#x2F;</a>

The python package Hypothesis[0] already does a great job bringing property-based testing to the people! I&#x27;ve used it and it&#x27;s extremely powerful.<p>[0]: <a href="https:&#x2F;&#x2F;github.com&#x2F;HypothesisWorks&#x2F;hypothesis">https:&#x2F;&#x2F;github.com&#x2F;HypothesisWorks&#x2F;hypothesis</a>

In the testing I focus on, testing of compilers, and in particular Common Lisp implementations, PBT has been invaluable, finding bugs in every implementation on which it was tried (I understand this is a universal experience in compiler testing, for example seen from Csmith on C compilers). Here, the property is often of the form &quot;these two related pieces of code should not crash and also compute the same thing&quot;. As a simplified example, an expression like<p>(+ x y)<p>should compute the same thing as<p>(the &lt;type1&gt; (+ (the &lt;type2&gt; x) (the &lt;type3&gt; y)))<p>where the types are randomly generated types that contain the values in question.

For structure generation I prefer Doug McIlroy&#x27;s approach: pick a tree size (from some arbitrary distribution), and then, of the <i>n</i> possible valid structures of that size, produce the <i>k</i>th one uniformly.<p><a href="https:&#x2F;&#x2F;www.cs.dartmouth.edu&#x2F;~doug&#x2F;nfa.pdf" rel="nofollow">https:&#x2F;&#x2F;www.cs.dartmouth.edu&#x2F;~doug&#x2F;nfa.pdf</a> gives an nfa variant; extending to a pda is an (interesting, I found) exercise.

I think property based testing becomes a lot easier when you can just use normal asserts like this: <a href="https:&#x2F;&#x2F;github.com&#x2F;unexpectedjs&#x2F;unchecked">https:&#x2F;&#x2F;github.com&#x2F;unexpectedjs&#x2F;unchecked</a>

Would be very valuable if someone could write a summary of novel ideas for practitioners (if there are any).

Property testing is a lot like formal methods - really cool, but almost entirely useless in ~95% of commercial contexts.<p>They&#x27;re both extremely useful when, say, building a parser, but when the kind of code you write involves displaying custom widgets, taking data and pushing it onto a queue, looking up data in a database, etc. integration tests have a lot more bang for the buck.

Just going to plug the excellent .NET PBT library, CsCheck [0]. I have used it quite a bit to excersize strange corners of my program logic and found several exotic bugs with it.<p>[0]: <a href="https:&#x2F;&#x2F;github.com&#x2F;AnthonyLloyd&#x2F;CsCheck">https:&#x2F;&#x2F;github.com&#x2F;AnthonyLloyd&#x2F;CsCheck</a>

&gt; My work is motivated by conversations with real PBT users, accentuating the benefits that they get from PBT and reducing the drawbacks.<p>ISTM that the benefits and drawbacks are just two sides of the same coin. The benefit is reduced testing effort. The drawback is that savings comes from sacrificing knowledge of exactly what has been tested.<p>&gt; I also observed that PBT users are not always good at evaluating whether their testing was effective.<p>Ah, there&#x27;s the rub. It seems that the benefit of making the tested property explicit is partially offset by not really knowing how much of the search space was explored automatically.

My main issue with that approach is that your tests become flaky due to random data generation in your tests. They can be fine in your staging build pipeline and fail at your prod build pipeline. Extensive logging doesn&#x27;t change that fact.

Nice work. I didn&#x27;t yet read it fully, but I love the idea. Looks to be a valuable thesis.

I read that as &quot;Property-Based Testing of People&quot;

LLMs are great at generating property-based tests.