I'm not an expert, but I believe I can answer some of the questions at the end of the article.<p>>are kprobes and ftrace the same kernel system? I feel like they are but I am confused.<p>They are not. kprobes and kretprobes have been around in the kernel for a lot longer than ftrace, and are exposed to multiple tracing programs. ftrace, perf, bpf, etc, all can make use of kprobes. (ftrace is pretty heavily dependent on kprobes for being useful, kprobes are still useful without ftrace)<p>>what is the relationship between perf and kprobes? if I just want to sample the registers / address of the instruction pointer from ls's execution, does that have anything to do with kprobes? with ftrace? I think it doesn't, and that I only need kprobes if I want to instrument a kernel function (like a system call), but I'm not sure.<p>Kprobes make a copy of the instruction you are probing, and replaces the first bytes of the instruction with a breakpoint instruction. When the CPU hits this, a trap occurs, and the registers are saved, and passed to the kprobe<p>Perf has the ability to collect CPU performance counters, tracepoints, kprobes, and uprobes. Tracepoints are added to the code of the application - they will include a definition in the header, and the actual tracepoint statement in the code itself. uprobes allow dynamic tracing of user level and library calls. You echo in a probe name, executable location, and offset, and then you can start tracing that probe.