The section on static analysis is somewhat damning:<p><i>Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has found a few bugs in SQLite, but those are the exceptions. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis.</i>
In 2010 I stumbled upn a bug that segfaulted Firefox:<p><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=581946" rel="nofollow">https://bugzilla.mozilla.org/show_bug.cgi?id=581946</a><p>The SQLite guy were ridiculously responsive and added all sorts of tests around the fix.<p>It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users!
The title says "how SQLite is tester", well I don't think the article talks about "how". It only talks about the different types of tests that they apply. I'm curious about "how" it is done.
Do you guys run a single "test" button and all the tests are executed? Or, all the tests are done independently? On which platforms do you test?
> Millions and millions of test cases<p>I wasn't sure how to interpret it, but then they mention that it's 40100 distinct test cases, parameterized. I would have rephrased it as "40K of parameterized test cases".
This seems like an extreme amount of tests, I would be very interested in hearing how they are managed. Without some sort of system around them I would imagine that many tests would lose their meaning.
Did I miss something, because I didn't see performance testing. Given some specific test bed, which has X amount of RAM, How much Insertion Per Second(IPS) I can get?
> The TH3 test harness is a set of proprietary tests, [...] TH3 is free to SQLite Consortium members and is available by license to others.<p>I think it's ridiculous that people make proprietary test suites. In my opinion, test suites shouldn't even be copyrightable let alone made proprietary. A test suite is used to define what the precise interface of an API is, it's just a piece of code rather than a standards document.<p>EDIT: To be clear, the actual test framework might be copyrightable. But the actual test cases should not be IMO. It's also a shame that a free software project requires using proprietary software in order to practically test it.