So you think you want to write a deterministic hypervisor?

Interesting project. I almost wish I had a concurrency bug to test it on.<p>&gt; Guest software running in the Antithesis platform still experiences concurrency similar to a multi-core &#x2F; multi-machine system, thanks to the process scheduling imposed by the guest OS<p>This might not exercise the full set of race conditions. When two threads are running simultaneously on separate cores (or hyper-threaded on the same core) they can interleave instructions at a much finer granularity than any OS time slicing would cause, even within instructions.<p>For example, could it find a race condition where two threads are executing INC [addr] on the same memory address, where context switching between instructions doesn&#x27;t trigger it?

I love this. Many a moon ago, I worked on a system called Aikido at MIT, which combined a special built hypervisor with a binary rewriting system (DynamoRio) to enable efficient time travel debugging and race detection of parallel applications.<p>If anyone&#x27;s interested, here&#x27;s a publication that talks about it in more detail:<p><a href="https:&#x2F;&#x2F;dspace.mit.edu&#x2F;handle&#x2F;1721.1&#x2F;72082" rel="nofollow">https:&#x2F;&#x2F;dspace.mit.edu&#x2F;handle&#x2F;1721.1&#x2F;72082</a><p>The use of performance counters here also reminds me of another project I worked on called Kendo, which was a posix thread like replacement that used performance counters to enforce a deterministic interleaving of synchronization operations (mutexes, etc). The system could guarantee determinism for programs that didn&#x27;t have race conditions. Back then, I found that counting instructions wasn&#x27;t deterministic on the processors of the time, but counting store operations was. If anyone&#x27;s interested in that work, here&#x27;s the publication:<p><a href="http:&#x2F;&#x2F;www.cag.csail.mit.edu&#x2F;~mareko&#x2F;asplos073-olszewski.pdf" rel="nofollow">http:&#x2F;&#x2F;www.cag.csail.mit.edu&#x2F;~mareko&#x2F;asplos073-olszewski.pdf</a>

What a tease! They describe in detail two problems they had to “invent workarounds” for, but say nothing about what the workarounds are. I’m very curious, since both of the problems sound quite hard to work around. I wonder if they’re being purposefully vague to make it harder for competitors to replicate their work…

I don&#x27;t understand how this works in the case of testing many applications running on many machines, where many services on many machines need to communicate with each other. We deploy a mix of systemd services and OCI containers (running on podman and Docker) to different machines, the exact mix on each machine depends on the machine&#x27;s intended purpose.<p>We currently run CI tests using QEMU VMs. These VMs comprise a few systems representative of those that we deploy to production.<p>Does adopting Antithesis mean that all non-containerized applications would need to be OCI-ified and every interaction would need to be mocked? There&#x27;s a sort of combinatorial explosion that I&#x27;m concerned about when I&#x27;m thinking about testing&#x2F;adding a new service to a system: All services on which it depends need to be mocked and all services which depend on it require creating a mocked version of it.<p>Seems like a lot of work. Can someone please help clarify things for me?<p>Also, how could we test the behavior of non-application code like drivers or the kernel itself?

I&#x27;m not familiar with the area, so am likely missing something, but how do they do deterministic thread-level context switching? Something like:<p><pre><code> var_1 = 0 var_2 = 0 thread_a: while true: something_complex() var_1 ++ thread_b: while true: something_complex() var_2 ++ </code></pre> Under the quoted definition of determinism, for every point in time, var_1 and var_2 should have the same values across all executions. But this would seem to amount to ensuring that exactly the same number of instructions are executed each time a thread is scheduled.

Hermit [1] is another cool effort at determinism&#x2F;reproducibility.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;facebookexperimental&#x2F;hermit">https:&#x2F;&#x2F;github.com&#x2F;facebookexperimental&#x2F;hermit</a>

How does this deal with non-determinism from the outside world? For example, let&#x27;s say one of my tests is flaky because it asks an external service to give it some data, and that external service is flaky in what it returns?<p>Or what if my bug is caused by bitflips in failing memory, that lead to impossible control flow paths being hit? Think something like:<p><pre><code> if x != 0: return 1&#x2F;x </code></pre> Failing with an error because x is 0.<p>Not hypothetical scenarios, both real bugs I&#x27;ve had to troubleshoot in my career.

Fun read. It has some similar ideas to <a href="https:&#x2F;&#x2F;dedis.cs.yale.edu&#x2F;2010&#x2F;det&#x2F;" rel="nofollow">https:&#x2F;&#x2F;dedis.cs.yale.edu&#x2F;2010&#x2F;det&#x2F;</a> but that was actually focused on multicore processing and the communication across cores.<p>Congrats on the launch.

I&#x27;ve long thought about these kind of OS designs, and what great features they can enable (such as time travel debugging). But the non-determinism introduced by inter-CPU interactions is a fundamental limitation, hence the need to run everything on a single isolated core.<p>One day(^TM) I&#x27;m really keen to design a multi-core CPU architecture that allows for deterministic message passing between cores in such a way that you could get this kind of software working with true parallelism.

This is a very promising project that I&#x27;ve seen a lot of attempts to do in the past, but never got to the level of progress that you have! Very impressive work!<p>I am sad that you decided to give up on solving the multi-core parallelism issue, since each guest running on a single core is a dead giveaway to malware that they&#x27;re not on a real machine, but it&#x27;s understandable. I do wonder if that means that some class of bugs will be undetectable to this hypervisor, though.

We do something similar in house where I work. Is it hard to onboard new customers? Since they make a special container for you they basically adopt your build system, which may be hard for them. Does this go beyond mutation testing?

Did you decide on a hypervisor instead of an emulator for overhead reasons? I don&#x27;t work there but I heard Microsoft has something called tkofuzz which similarly emphasizes determinism but uses Bochs internally.

This is for a distributed database product that claims to bypass the CAP theorem or have I misunderstood?<p>&gt; Back then Spanner wasn’t public yet and a lot of people misinterpreted the CAP theorem to say that a strongly consistent database couldn’t also be highly available in the face of network faults.<p>- <a href="https:&#x2F;&#x2F;antithesis.com&#x2F;blog&#x2F;is_something_bugging_you&#x2F;" rel="nofollow">https:&#x2F;&#x2F;antithesis.com&#x2F;blog&#x2F;is_something_bugging_you&#x2F;</a>

strange they seem unaware of the RR deterministic debugger work<p>especially as that builds off of the extensive work done on x86 counter determinism here: <a href="https:&#x2F;&#x2F;web.eece.maine.edu&#x2F;~vweaver&#x2F;projects&#x2F;deterministic&#x2F;" rel="nofollow">https:&#x2F;&#x2F;web.eece.maine.edu&#x2F;~vweaver&#x2F;projects&#x2F;deterministic&#x2F;</a><p>it turns out x86&#x2F;amd chips many of the perf counter events are offset by the (unpredictable) interrupt count because the interrupt return instruction uop gets counted as both a user and kernel instruction. On many processors the retired store instruction avoids this issue.