Running three hours of Ruby tests in under three minutes

We&#x27;re not nearly at Stripe&#x27;s scale, but my startup (Spreemo) has achieved pretty amazing parallelism using the commercial SaaS CircleCI. We have 3907 expects across 372 RSpec and Cucumber files. Our tests complete in ~14 minutes when run across 8 containers.<p>One of the great strengths for CircleCI is that they auto-discover our test types, calculate how long each file takes to run, and then auto-allocate the files in future runs to try to equalize the run times across containers. The only effort we had to do was split up our slowest test file when we found that it was taking longer to complete than a combination of files on the other machines.<p>I also like that I can run pronto <a href="https:&#x2F;&#x2F;github.com&#x2F;mmozuras&#x2F;pronto" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mmozuras&#x2F;pronto</a> to post Rubocop, Rails Best Practices, and Brakeman errors as comments on Github.

I&#x27;m super curious how Stripe approaches end-to-end testing (like Selenium&#x2F;browser testing, but maybe something more bespoke too)<p>My understanding is that they have a large external dependency (my term: &quot;the money system&quot;), and running integration tests against it might be tricky or even undependable. Do they have a mock banking infrastructure they integrate against?

I am tired of this technology having to be re-invented time and time again.<p>The best I ever saw was an internal tool at Microsoft. It could run tests on devices (Windows Mobile phones, but it really didn&#x27;t care), had a nice reservation and pool system and a nice USB--&gt;Ethernet--&gt;USB system that let you route any device to any of the test benches.<p>This was great because it was a heterogeneous pool of devices, with different sets of tests that executed appropriately.<p>The test recovery was the best I&#x27;ve ever seen. The back end was wonky as anything, every single function returned a BOOL indicating if it had ran correctly or not, every function call was wrapped in an IF statement. That was silly, but the end result was that every layer of the app could be restarted independently, and after so many failures either a device would be auto removed from the pool and the tests reran on another device, or a host machine could be pulled out, and the test package sent down to another host machine.<p>The nice part was the simplicity of this. All similar tools I&#x27;ve used since have involved really stupid setup and configuration steps with some sort of crappy UI that was hard to use en-masse.<p>In comparison, this test system just tool a path to a set of source files on a machine, the compilation and execution command line, and then if the program returned 0 the test was marked as pass, if it returned anything else it was marked as fail.<p>All of this (except for copying the source files over) was done through an AJAX Web UI back in 2006 or so.<p>Everything I&#x27;ve used since than has either been watching people poorly reimplementing this system (frequently with not as good error recovery) or just downright inferior tools.<p>(For reference a full test pass was ~3 million tests over about 2 days, and there were opportunities for improvement, network bandwidth alone was a huge bottle neck)<p>All that said, the test system in the link sounds pretty sweet.

If you want to implement this locally without using mini-test checkout test-queue by Aman Gupta at github.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;tmm1&#x2F;test-queue" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tmm1&#x2F;test-queue</a><p>One thing that really sped up our test suite was by creating an NGINX proxy that served up all the static files instead of making rails do it. This saved us about 10 minutes off our 30 minute tests.

Very cool stuff. For reference at GitLab we use a less impressive and simpler solution. We split the jobs off in <a href="https:&#x2F;&#x2F;gitlab.com&#x2F;gitlab-org&#x2F;gitlab-ce&#x2F;blob&#x2F;master&#x2F;.gitlab-ci.yml" rel="nofollow">https:&#x2F;&#x2F;gitlab.com&#x2F;gitlab-org&#x2F;gitlab-ce&#x2F;blob&#x2F;master&#x2F;.gitlab-...</a> These jobs will be done by separate runners, this brought our time down from 1+ hours to 23 minutes <a href="https:&#x2F;&#x2F;ci.gitlab.com&#x2F;projects&#x2F;1&#x2F;refs&#x2F;respect_filters&#x2F;commits&#x2F;e58e75aa8860c4c1530ebe7ad1e4bf557fa1e848" rel="nofollow">https:&#x2F;&#x2F;ci.gitlab.com&#x2F;projects&#x2F;1&#x2F;refs&#x2F;respect_filters&#x2F;commit...</a>

How much cheaper (in time, code, effort, complexity) would it be if:<p>- Their language runtime supported thread-based concurrency, which would drastically reduce implementation complexity <i>and</i> actual per-task overhead, thus improving machine usage efficiency AND eliminating the concerns about managing process trees that introduces a requirement for things like Docker.<p>- Their language runtime was AOT or JIT compiled, simply making <i>everything</i> faster to a degree that test execution could be reasonably performed on <i>one</i> (potentially large) machine.<p>- They used a language with a decent type system, significantly reducing the number of tests that had to be both written and run?

One thing I&#x27;ve noticed since coding with immutable data structures &amp; functions (rather than mutable OOP programs) is how tests run really fast, and are easy to run in parallell.<p>I&#x2F;O only happens in a few functions, and most other code just takes data in -&gt; transforms -&gt; returns data out. This means I only have few functions that need to &#x27;wait&#x27; on something outside of itself to finish, and much lesser delays in the code.<p>This is coding in Clojure for me, but you can do that in any language that has functions (preferable with efficient persistent data structures. Like the tree-based PersistentVector in Clojure).

Love this. Sometimes testing can be a huge pain in the ass. I know more than one project I work on where getting them to run is a lot of effort in itself.<p>There is something to be said about code quality and having tests run in under a few seconds. The ideal situation is when you can have a barrage of tests run as fast as you are making changes to code. If we ever got to the point of instant feedback that didn&#x27;t suck I&#x27;d think we&#x27;d change a lot about how we think about tests.

<i>We opted for an alternate, dynamic approach, which allocates work in real-time using a work queue. We manage all coordination between workers using an nsqd instance... In order to get maximum parallel performance out of our build servers, we run tests in separate processes, allowing each process to make maximum use of the machine&#x27;s CPU and I&#x2F;O capability. (We run builds on Amazon&#x27;s c4.8xlarge instances, which give us 36 cores each.)</i><p>This made me long for a unit test framework as simple as:<p><pre><code> $ make -j36 test </code></pre> Where you&#x27;ve got something like the following:<p><pre><code> $ find tests&#x2F; tests&#x2F;bin&#x2F;A tests&#x2F;bin&#x2F;B ... tests&#x2F;input&#x2F;A tests&#x2F;input&#x2F;B ... tests&#x2F;expected&#x2F;A tests&#x2F;expected&#x2F;B ... tests&#x2F;output&#x2F; $ cat Makefile test : $(shell find tests&#x2F;bin -type f | sed -e &#x27;s@&#x2F;bin&#x2F;@&#x2F;output&#x2F;@&#x27;) tests&#x2F;output&#x2F;% : tests&#x2F;bin&#x2F;% tests&#x2F;input&#x2F;% tests&#x2F;expected&#x2F;% @ printf &quot;testing [%s] ... &quot; $@ @ sh -c &#x27;exec $$0 &lt; $$1&#x27; $^ &gt; $@ @ # ...runs tests&#x2F;bin&#x2F;% &lt; tests&#x2F;input&#x2F;% &gt; tests&#x2F;output&#x2F;% @ sh -c &#x27;exec cmp -s $$3 $$0&#x27; $@ $^ &amp;&amp; echo pass || echo fail @ # ...runs cmp -s tests&#x2F;expected&#x2F;% tests&#x2F;output&#x2F;% clean : rm -f tests&#x2F;output&#x2F;* </code></pre> You get test parallelism and efficient use of compute resources &quot;for free&quot; (well, from make -j, because it already has a job queue implementation internally). This setup closely resembles the &quot;rts&quot; unit test approach you&#x27;ll find in a number of djb-derivative projects.<p>The defining obstacle for Stripe seems like Ruby interpreter startup time though. I&#x27;m not sure how to elegantly handle preforked execution in a Makefile-based approach. Drop me a line if you have ideas or have tackled this in the past, I&#x27;ve got a couple projects stalled out on it.

On a previous project, I had built a shell script that essentially created n mysql databases and just distributed the test files under n rails processes.<p>We were able to run tests that took an hour in about 3 minutes. It was good enough for us. Nothing sophisticated for evenly balancing the test files, but it was pretty good for 1-2 days of work.

&quot;This second round of forking provides a layer of isolation between tests: If a test makes changes to global state, running the test inside a throwaway process will clean everything up once that process exits.&quot;<p>But, then how do you catch bugs where shared mutable state is not compatible with multiple changes?

Congratulations on what look a very challenging task. I&#x27;m assuming a part of those tests hit a database. How have you dealt with it? I assume that a single instance, even on a powerful bare server could be a road blocker in this situation. A few insights on the Docker&#x2F;Containerization part of it would also be nice!

This is an interesting and possibly overlooked problem with using slow languages like Ruby - your unit tests take forever to run. (unless you spend a lot of engineering effort on making them run faster, in which case they may run somewhat acceptably fast)

I guess a lot of problems come from the stupidly brain dead way people usually write tests (because it&#x27;s the &quot;recommended TDD way&quot;)<p>Things like using the same setup function for every test and setting up&#x2F;tearing down for every test regardless of dependencies<p>Also tests like<p><pre><code> def test1(): do_a() check_condition_X() </code></pre> then<p><pre><code> def test2(): do_a() check_condition_Y() </code></pre> Or<p><pre><code> def test1(): do_a() check_condition_X() def test2(): do_a() do_b() check_condition_Y() </code></pre> When it could have been consolidated into 1 test<p>Then people wonder why it takes so much time?<p>Also helpful is if you can shutdown database setup for tests that don&#x27;t need it

Oh hey, we have the same sort of system here. It&#x27;s 60,000 Python tests which take ~28 hours if run serially, but we keep it around 30-40 minutes. We wrote a UI &amp; scheduler &amp; artifact distribution system (which we&#x27;re probably going to replace with S3). We run selenium &amp; unit tests as well as the integration tests.<p>We&#x27;ve noticed that starting and stopping a ton of docker containers in rapid succession really hoses dockerd, also that Jenkins&#x27; API is a lot slower than we expected for mostly-read-only operations.<p>Have you considered mesos?

Not sure if this has already been answered, but would Stripe&#x27;s methods only work with unit tests where tests are not dependent on each other?<p>How would one go about building a similar distributed testing setup for end-to-end tests where a sequence of tests have to be run in particular order. Finding the optimal ordering &#x2F; distribution of tests between workloads would certainly be more complicated. Maybe they could be calculated with directed graph algorithms?

<i>needle scratching on record</i><p>They have an average of 9 assertions per test case. I think I may see part of their problem.

Any reason why a financial infrastructure provider like Stripe would run CI tests on someone elses infrastructure? Isn&#x27;t that a no go from a security point of view? Or - how do you trust the hosted CI company not to look at your code?

I wrote a rubygem called cloudspeq (<a href="http:&#x2F;&#x2F;github.com&#x2F;meesterdude&#x2F;cloudspeq" rel="nofollow">http:&#x2F;&#x2F;github.com&#x2F;meesterdude&#x2F;cloudspeq</a>) that distributes rails rspec spec&#x27;s across a bunch of digital ocean machines to reduce test execution time for slow test suits in dev.<p>one of the things I did that may be of interest is to break up spec files themselves to help reduce hotspots (or dedicate a machine to it specifically)<p>Not as complex or as robust as what they did, but it works!

It&#x27;s interesting to imagine, for a test suite that would take three hours, how much of the execution time is state management vs algorithm execution.

<a href="http:&#x2F;&#x2F;engineering.imvu.com&#x2F;2011&#x2F;01&#x2F;19&#x2F;buildbot-and-intermittent-tests&#x2F;" rel="nofollow">http:&#x2F;&#x2F;engineering.imvu.com&#x2F;2011&#x2F;01&#x2F;19&#x2F;buildbot-and-intermit...</a>

Does this mean each process has its own database or are you able to use transactions with the selenium&#x2F;capybara tests?

Pull-based load balancing is a generally underrated technique.

&gt; Initially, we experimented with using Ruby&#x27;s threads instead of multiple processes<p>Why, to be cool? Tests are a classic case of things that should be run in isolation - you don&#x27;t want tests interfering with earth other or crashing the whole test suite. Using separate processes would have been the sensible approach to start with.

Was anyone else expecting the article to be about replacing Ruby with a compiled language?