The sad state of test frameworks

I have many bad experiences with DIY test frameworks. Please think twice about doing this, especially for something others might use. The problems are analogue to developing software without best practices.<p>At some point, that simple framework won&#x27;t do what you need it to, so it grows. And grows. Specificity and &quot;clever&quot;&#x2F;&quot;smart&quot; frameworks are initially awesome to avoid boilerplate. But the more specific you make it, the quicker you hit these limits. The more clever you make it, the less comprehensible and more brittle the tests are. Lack of documentation is always a problem, as is predicting what such a framework needs to support. For the most part, it will just end up an unmaintained mess that someone down the line will have to throw out, and rewrite all the tests.<p>For acceptance testing, RobotFramework seems like overkill, and the syntax is weird. But it&#x27;s a super effective generic testing framework, and suits the needs of thousands of projects. It&#x27;s not perfect and you&#x27;ll probably never love it, but it&#x27;ll get the job done, no matter how big the project.<p>I agree with the bad state of C unit test frameworks. We had to figure out how to add unit testing to C code, and in the end we just went with Google Test and all the headache that C++ brings with it. That was a while ago. After a quick search, I think I&#x27;d give µnit a try first.

For C and C++, what&#x27;s wrong with using assert() in an integration_test() function or many &lt;func&gt;_test() functions, called when passed a --test flag or compiled as a separate binary?<p>For me, benchmarking isn&#x27;t really useful unless it&#x27;s a profiler in a real world use situation. Otherwise you have no idea how often each API function is called.

Is this about the sad state of test frameworks or about the sad state of C test frameworks? There&#x27;s a reference to JUnit I don&#x27;t understand. Junit provides the ability to aggregate tests in suites (strict hierarchies wtf?).<p>Benchmark may be a different concern.. I know you can do something with rules but it may require more time to set up.<p>By the way: creating a small, self developed tool that just does what you need is sometimes a good idea, but it doesn&#x27;t mean everything else is shit. Any library serving a large user base will serve somebody better than others.

I use a roll-yer-own test framework that (I believe) strikes the perfect balance between setup costs, simplicity and flexibility. It&#x27;s a small utility that just looks through a given directory tree, finds folders called *&#x2F;test, compiles and runs each .cpp file in that folder (taking cues from formatted comments inside the file if additional files are required), and compares the output to a stored &#x27;good&#x27; output for that file, flagging an error if the files differ.<p>When developing code, you&#x27;ll almost always end up writing some ad-hoc testing code anyway: Generate some input data, call the code you&#x27;re testing, printf() some results. At development time, you read this output to verify your code&#x27;s working well. My system just makes this a bit more methodical and saves a &#x27;known good&#x27; result - if the output changes then the code may be wrong. Once you&#x27;ve fixed the issue (or determined that the new behaviour is in fact correct) you update the known-good output.<p>Dead simple, basically maintains itself, still catches any problems that the tests would catch.

Interesting - I came to a similar conclusion from a completely different angle and blogged about it the other day:<p><a href="https:&#x2F;&#x2F;zwischenzugs.wordpress.com&#x2F;2017&#x2F;03&#x2F;18&#x2F;clustered-vm-testing-how-to&#x2F;" rel="nofollow">https:&#x2F;&#x2F;zwischenzugs.wordpress.com&#x2F;2017&#x2F;03&#x2F;18&#x2F;clustered-vm-t...</a><p>I&#x27;ve similarly rolled my own bash-based solutions, as testing is really as simple as running a script and getting a &#x27;0&#x27; or non-zero exit code.

I rolled my own a while back <a href="https:&#x2F;&#x2F;github.com&#x2F;keithn&#x2F;seatest" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;keithn&#x2F;seatest</a><p>I kept it very vanilla C and avoided having to maintain a suite structure in favour of a function call system which can be used on embedded systems without much resources.<p>I should probably do more to maintain that project and improve it as I have outstanding PRs it seems!

I use Google Test, which integrates nicely with CLion and Jenkins.

I haven&#x27;t documented fully finished work on it yet (but its now running and working for me), but apparently like everyone else I wasn&#x27;t very impressed with testing frameworks, so decided to attempt to roll my own for Common Lisp rather than C:<p>See <a href="https:&#x2F;&#x2F;github.com&#x2F;DJMelksham&#x2F;testy&#x2F;tree&#x2F;master" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;DJMelksham&#x2F;testy&#x2F;tree&#x2F;master</a> if anyone is interested, though I&#x27;m not sure how useful it will be at this stage.<p>Like the article author, I settled on tags for tests rather than explicit hierarchies.<p>Also somewhat like the author, I care about benchmarks to a degree so amongst other things, each test keeps its latest timing and results stats: and because tests serialise to text&#x2F;lisp code, each test-code&#x2F;test-run can be version controlled with a project.<p>My project has some slight diversions&#x2F;additions to the article:<p>1. Being able to capture previously evaluated code and returned results from the REPL and automatically package up the form and the result as a test. Obviously not as applicable to C, but i found myself interactively trying to get a function to work then capturing the result upon success as a regression test rather than the usual &quot;design test up front&quot; from TDD (although that&#x27;s possible too).<p>2. I had a long long internal debate about the philosophy of fixtures. The concept of environments&#x2F;fixtures in which you could run a series of tests was in fact my initial plan, and although I&#x27;ve backed out of it, I think I could theoretically add such a thing back in, but I&#x27;m not sure I want to now...<p>It seems to me by supplying fixtures&#x2F;environments in which you run multiple tests, you gain &quot;not paying the setup cost&#x2F;don&#x27;t repeat yourself&quot; and &quot;flexibility of running tests in arbitrary environments&quot;. What I considered the cost was the tendency to move away from true test independence (the benefit of which is easier naive multi-threading of running the test suite), and no longer a full referential transparency&#x2F;documentation of a test. Test 2784 says it failed in the last run. Is that because I ran it in context A or context B...and what else was running with it and what caused the failure? What is the performance&#x2F;timing actually measuring? Sort of like lexical scoping, I wanted the reason for the test&#x27;s values and failures to be as much as possible &quot;in the test source&quot; and no where else.<p>This philosophy of mine creates obvious limitations: to try to get around them a bit, while keeping the benefits, I made two more design choices.<p>3. Composable tests: New tests can be defined as the combination of already existing tests.<p>4. Vectorise test functions!: Most important functions work on vectors or sets of tests as well as individual tests. The function (run-tests), for instance runs a vector of tests passed in to it. The functions (all-tests), (failed-tests), (passed-tests), (get-tests-from-tags), etc, all return vectors of tests you would expect from their names which can then be passed on to the other functions that map across vectors of tests. (print-stats) for example can print statistics on the entire test-suite (because by default its input is the result of the function (all-tests)), but it also accepts any arbitrary set of tests, be it passed-tests, failed-tests or a particular set marked by a specific tag. And because each test is self-contained, all results&#x2F;reports can just be generated by just combining the internal results&#x2F;reports of each individual test in the vector. Copy a set of old tests as a basis for new ones, compose multiple tests or functions into one, and&#x2F;or map new settings or properties across an arbitrary group of tests.<p>Anyway, I&#x27;m curious to hear other people&#x27;s experiences and design choices in this regard.