<i>It writes an invalid instruction at this location. What ever this instruction, it just has to be invalid.</i><p>On x86 at least, it is a valid instruction. INT3, or CC in hex. There are also the debug registers which implement breakpoints without modifying any code, although it's limited to a maximum of 4 at once.<p>Characterising gdb as a "C debugger" is quite appropriate --- try to debug the Asm directly with it is an excruciating experience.
See also the "Writing a Linux Debugger" series of posts in which a source-level debugger is implemented: <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/" rel="nofollow">https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/</a>
This is a fantastic summary of debugger implementation!<p>Another great one that actually walks through writing a basic debugger is Eli Bendersky's series[1].<p>One nitpick:<p>> It could, and that would work (that the way valgrind memory debugger works), but that would be too slow. Valgrind slows the application 1000x down, GDB doesn't. That's also the way virtual machines like Qemu work.<p>This is usecase-dependent: running a program until you hit a breakpoint will be significantly faster with `int 3`, but running a piece of instrumentation on every instruction (or branch, or basic block, or ...) will be significantly faster with Valgrind (or another dynamic binary instrumentation framework). This is because Valgrind and other DBI tools can rewrite the instruction stream to sidecar instrumentation into the same process, versus converting every instruction (or other program feature) into a sequence of expensive system calls.<p>[1]: <a href="https://eli.thegreenplace.net/tag/debuggers" rel="nofollow">https://eli.thegreenplace.net/tag/debuggers</a>
One useful reference for this is <a href="https://www.cs.tufts.edu/~nr/pubs/retargetable-abstract.html" rel="nofollow">https://www.cs.tufts.edu/~nr/pubs/retargetable-abstract.html</a>
It writes an invalid instruction at this location. What ever this instruction, it just has to be invalid.<p>RISC-V actually did this in a special instruction called ebreak. It can change the CPU privileged mode into Debug Mode.