rr – record and replay debugger for C/C++

I've used rr very sucessfully for reverse engineering a large code base using a break on variable change combined with reverse-continue. Took the time to extract critical logic way down.

Perhaps worth mentioning is that someone attempted to port this to Rust and got about 60,000 lines of code into it before archiving the project. I feel like comparing these two efforts would be an interesting case study on the impacts &#x2F; benefits &#x2F; limitations or difficulties, etc involved in rewriting from C++ to Rust.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;sidkshatriya&#x2F;rd&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;sidkshatriya&#x2F;rd&#x2F;</a>

Is it truly only for C&#x2F;C++?<p>My limited understanding says a debugger needs: a list of symbols (.pdb files on windows, can&#x27;t remember what they are on linux), understanding of syscalls and a few other similar things. I thought they don&#x27;t care too much what generated the binaries they are debugging (obviously as long as it&#x27;s native code).<p>Doesn&#x27;t rr work with other languages like rust, zig, odin, nim, and similar ones? Obviously, I wouldn&#x27;t expect it to work for python, js, c# and other languages with managed memory.

rr is really cool, but almost every time I have decided to pull it out as one of the &quot;big guns&quot; it turns out that I have a concurrency bug and so rr is unable to reproduce it.<p>Despite that, it would be very, very, very cool if some languages built rr directly into their tooling. Obviously you can always &quot;just&quot; use rr&#x2F;gdb, but imagine if rr invocations were as easy to set up and do as pdb is in Python!

See also <a href="https:&#x2F;&#x2F;pernos.co&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pernos.co&#x2F;</a> which is based on rr but adds a queryable database of the whole program execution, which allows you to do things like this:<p>&gt; [...] just click on the incorrect value. With full program history Pernosco can immediately explain where this value came from. The value is tracked backwards through events such as memcpys or moves in and out of registers until we reach a point where the value &quot;originated&quot; from, and each step in this process is displayed in the &quot;Dataflow&quot; panel that opens automatically. There is no need to read or understand the code and think about what might have happened, we can simply ask the debugger what did happen.<p><a href="https:&#x2F;&#x2F;pernos.co&#x2F;examples&#x2F;race-condition" rel="nofollow">https:&#x2F;&#x2F;pernos.co&#x2F;examples&#x2F;race-condition</a>

GDBs built-in reverse debugging: <a href="https:&#x2F;&#x2F;www.sourceware.org&#x2F;gdb&#x2F;wiki&#x2F;ProcessRecord&#x2F;Tutorial" rel="nofollow">https:&#x2F;&#x2F;www.sourceware.org&#x2F;gdb&#x2F;wiki&#x2F;ProcessRecord&#x2F;Tutorial</a><p>I assume <i>rr</i> provides more features and flexibility. Anyway I want to mention that GDB itself can already reverse debug for some time now.

On Windows you can use WinDbg for the same thing. It has better support for debugging multi-threaded issues.<p><a href="https:&#x2F;&#x2F;www.forrestthewoods.com&#x2F;blog&#x2F;windbg-time-travelling-debugger-is-amazing-magic&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.forrestthewoods.com&#x2F;blog&#x2F;windbg-time-travelling-...</a>

Long ago, VMWare workstation supported doing this, but not just for userspace programs but also for kernels and even drivers, in a VM. The feature shipped and existed for a few versions before it was killed by internal politics.

Are rr’s problems with Ryzen CPUs now firmly in the past or not?

A couple of previous discussions:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31617600">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31617600</a> (June 2022)<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18388879">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18388879</a> (Nov 2018)

I used this to help make my toy JIT compiler: <a href="https:&#x2F;&#x2F;github.com&#x2F;danthedaniel&#x2F;BF-JIT">https:&#x2F;&#x2F;github.com&#x2F;danthedaniel&#x2F;BF-JIT</a><p>Super useful, especially considering I know barely anything about x86-64.

I almost use rr every day, along with a gdb frontend: cgdb.<p>rr record &#x2F;tmp&#x2F;Debug&#x2F;bin&#x2F;llvm-mc a.s &amp;&amp; rr replay -d cgdb<p>I&#x27;ve have success story with some bugs only reproducible with LTO. Without rr it would be a significant challenge.<p>It would be nice if Linux kernel could be debugged with rr. Does anyone have success with kernel under rr+qemu ? :)

[off-topic] does anyone here who regularly uses a debugger (even just breakpoints and watchers in their IDE) use it for async execution? I&#x27;ve never tried, but I&#x27;m just trying to think how all that jumping around the executor and any runtime would work (if at all).

Is it possible to use this with C&#x2F;C++ code compiled to dll&#x2F;so and called by Python?

Curious if anyone tried rr for&#x2F;on Android? It seems possible to crosscompile it and it could be a good tool for native side debugging

Has anyone gotten rr to work with opengl or vulkan? It seems to always crash for me after making an opengl call.

Can someone explain how it works?