How Rust is tested

I love the concept of the &quot;crater run&quot; (or I guess now &quot;cargo bomb&quot;?), where the entire crate (i.e. &quot;package&quot;) ecosystem is compiled, as a test for speculative compiler changes.<p>It&#x27;s one of the huge benefits of being both a statically-typed compiled language and offering a modern package system. A lot of languages have the types &amp; compiling but no standard package ecosystem (Haskell, Go, Java), or the package ecosystem but not the compiling (Ruby+Gems, JS+npm, etc). Rust is the only language I can think of that has both, although I&#x27;m sure there are others. I&#x27;m not sure if there&#x27;s tooling for compiler writers in other languages to compile them all, though.<p>And this feature will only get more and more powerful as the ecosystem grows. I think it&#x27;s a seriously important tool to keep the rust language feeling stable.

&gt; There’s a big downside though in that landing patches to Rust is serialized on running the test suite on every patch, and it takes a particularly long time to Run the Rust test suite in all the configurations we care about.<p>One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (&gt;40 merges&#x2F;day) and ~1hr maximum test times.<p>Given a queue of patches A, B, C, D, start the normal testing against A, but <i>also</i> start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it&#x27;s still &quot;compatible&quot; with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn&#x27;t cause any slowdowns versus the fully serialized case.<p>It looks like bors already supports something like this with &quot;rollups&quot;, where simple fixes can be marked for testing together: <a href="https:&#x2F;&#x2F;internals.rust-lang.org&#x2F;t&#x2F;batched-merge-rollup-feature-has-landed-on-bors&#x2F;1019" rel="nofollow">https:&#x2F;&#x2F;internals.rust-lang.org&#x2F;t&#x2F;batched-merge-rollup-featu...</a>

&gt; At some point though LLVM began doing a valid optimization that valgrind was unable to recognize as valid, and that made its application useless for us, or at least too difficult to maintain.<p>What optimization was that? Some searching didn&#x27;t turn up any relevant results (other than this article).

&gt; The thing we do differently from most is that we run the full test suite against every patch, as if it were merged to master, <i>before</i> committing it into the master branch, whereas most CI setups test <i>after</i> committing, or if they do run tests against every PR, they do so before merging, leaving open the possibility of regressions introduced during the merge.<p>Hm, Travis CI on GitHub runs tests on a pull request before the merge. Is what they&#x27;re doing really that unusual?

I appreciate the thoroughness of this. I&#x27;ve been using Rust for 2 years now, and the compiler has been rock solid.

It would be really nice if Travis had Linux-on-ARMv7 and Linux-on-aarch64 options on real ARM hardware. I wonder what the economics of Travis building such a thing on top of an existing ARM cloud offering like Scaleway would be.

&gt; Today the longest-running configuration takes over 2 hours.<p>I&#x27;m curious which configuration that is, and how it could be accelerated.

I have been taking inspiration from the Rust project and the way they go about things in general. In my workplace, we have a bot similar to bors, which does the merge request testing and is the gatekeeper.<p>Tools like bors act as a &quot;force-multiplier&quot;.

I dream with the day we will have a GCC Rust compiler.

Heads up that your Google fonts (in site.css) trigger a warning about insecure scripts, and in Chrome at least they therefore don&#x27;t load by default.

Anyone care to enlighten me why fuzz testing and not mutation testing?

I don&#x27;t get it, because you have just explained how to unit test everything else. Which is way better than nothing. Not sure how it is moot, also not sure how it is related to the article :)

Unit test in a nutshell, for a business system:<p>Practically every business system consists of a view, the logic and the database, plus some webservice and whatnot, but mainly its those 3 parts (the so called 3 layers).<p>Can you unit test, test the view layer? Not really. Can you unit test, test the database by doing real test? Again, not really and its not always possible. So, Unit test is mainly focused in the layer in between of the view and database and both, by their nature, can&#x27;t be automatically tested.<p>I like the idea of Unit Test but, for business system, its moot.