Small, somewhat nit-picky critique: the man pages for system calls are in section 2. If you want to see the docs for the "read()" syscall, and not the bash builtin "read", saying "man read" w̶o̶n̶'̶t̶ may not (see follow-up) do what you expect. Instead, you should say<p><pre><code> man 2 read
</code></pre>
This should probably be mentioned somewhere.<p>Otherwise, great writeup. Thanks for sharing!<p>(edited)
Don't forget it's userspace equiv (strace is syscalls), ltrace. This tracks all lib calls made by process.<p>Under windows, strace is an SSL/TLS monitoring tool (also hella useful). It shows payloads passed to CryptoAPI/CNG libs so you can easily troubleshoot explicitly encrypted protocols like ldaps. Especially useful if you use client authenticated TLS where is is not possible to use a TLS mitm proxy to snoop the layer 7 data.
Mac OS X has a suite of tools built on a similar package called dtrace—opensnoop and execsnoop. Gives really nice real time lists of all files opened on the system and all binaries executed, respectively.
Thanks for the writeup! strace should definitely be in your toolbox. There is also systemtap, which I like a lot as well. It has some problem on Linux though, especially only being widely supported of Linux > 3.5 if the distro you are using does not ship with patches. Custom userspace probes are a real strong point.<p>I wrote a short article about stap using Rubys probes as an example: <a href="http://www.asquera.de/blog/2014-01-26/stap-and-ruby-2" rel="nofollow">http://www.asquera.de/blog/2014-01-26/stap-and-ruby-2</a>
You can use Process monitor
<a href="http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx" rel="nofollow">http://technet.microsoft.com/en-us/sysinternals/bb896645.asp...</a>
to see a similar overview of low level activity. You won't see all the system calls, you can't pipe the output directly, but there is a UI and you don't have to look up file descriptors
Has anyone heard of a program that will take strace (or dtrace) output and create a pretty diagram showing which commands call which commands and which files they read or create?<p>We've got a fairly complicated bioinformatics pipeline that calls about 100 other programs, and creates or reads about 100 different files. I'd love a way to create a picture of what's going on. Which files each program uses, etc.<p>If such a program doesn't exist, would that be worth building? Could it be something I could potentially sell?
"perf top -e syscalls:<i>statfs</i>"<p>particularly when you don't know which process is calling all the syscalls.<p>Mix "perf record" and "perf trace" & you have the next generation of strace tools.
I use strace all the time doing ops at Crittercism. Some of the random things it's helped with/taught me:<p>- allowed exploring forking behavior of daemons, in particular the nitty-gritty of gunicorn's prefork behavior, and understanding the rationale behind single- and double-fork daemons generally (very important to understand for job control e.g. writing upstart/init.d jobs)<p>- isolated hot reads to memcache in situ, by identifying the socket associated with the memcache connection, and finding which key was read the most by a process (we built better logging after the fact, but sometimes there's no substitute for instrumenting prod during tough perf/stress problems)<p>- let me explore the behavior of node.js's several threads, and find one of them sending "X" over a socket to the other (still not quite sure what this is, some kind of heartbeat/clock tick?)<p>- helped understanding "primordial processes" and the exact details of how forking/reparenting work on linux<p>It's a great tool and one that every ops/infrastructure engineer should be familiar with.
Quick strace command that I use all the time to see what files a process is opening:<p><pre><code> strace -f <command> 2>&1 | grep ^open
</code></pre>
Really useful to see what config files something is reading (and the order) or to see what PHP (or similar) files are being included.<p>There's normally other ways to do this (eg using a debugger) but sending strace's stderr to stdout and piping through grep is useful in so many cases it's become a command I use every day or 2.
Strace is easy to use, commonly available, and very useful in many situations.<p>More modern tools such as dtrace for the solaris and systemtap for linux addresses similar problems but with a broader coverage.
Also check out ltrace... Shows the calls to other libraries the process is making...<p>I'd also like to point out that a key to using strace successfully is the result column... Programs that fail often make system calls that fail right before they exit... You can often tell what the program is trying and failing to accomplish...
In case you're curious, this is how ltrace (strace's library equivalent) works: <a href="http://www.opersys.com/blog/ltrace-internals-140120" rel="nofollow">http://www.opersys.com/blog/ltrace-internals-140120</a>
I’ve used strace before to help diagnose issues with buggy software I was using and I thought this was a great article.<p>I just thought I’d let people know that it can be a lot easier to read strace’s output if you read the output log file using Vim as it contains a syntax file which can highlight PIDs, function names, constants, strings, etc. Alternatively, if you don’t want to create an strace log file, you could pipe the output to Vim and it will automatically detect it as being strace output, e.g.<p><pre><code> strace program_name 2>&1 | vim -</code></pre>
Totally agree that strace is an awesome tool. I've even used it with Java apps that were behaving wierdly, just attach and see what it is saying to the kernel.
Bending to the will of the people, I have appended a conclusion, clarifying the fate of the Lotus Domino server. <a href="http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/" rel="nofollow">http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/</a>
Just a few hours ago, a newly minted Ubuntu binary was crashing due a library version mismatch. I thought I had updated the shared libraries to point to the new versions. But definitely something was still hooked to the old version. I just couldn't figure out how/where. ldd wasn't of much help because everything was OK according to it. "If only I can get a bit more info when the binary is running and spit out everything before the crash."<p>Tried my luck with gdb. Sure enough...there was libQt5DBus pointing to the old libs leading to the crash. If you are feeling particularly adventurous, you can step one instruction at a time after starting. Even without debug symbols, there is quite a lot of info that be used while troubleshooting.
There's a lot of fun to be had with strace. I wrote a tiny perl script that spies on the file descriptors of another process and outputs it to your terminal: <a href="https://github.com/psypete/public-bin/blob/public-bin/src/system/dumpfd.pl" rel="nofollow">https://github.com/psypete/public-bin/blob/public-bin/src/sy...</a>
It's instructive to see how much simpler the strace output for a simple program is when the program is statically linked. Especially if you use an alternative libc like musl (<a href="http://musl-libc.org/" rel="nofollow">http://musl-libc.org/</a>).
Don't forget that sometimes strace is overkill, and similar more easily parsed things can be used instead, for example, /usr/bin/time (vs bash time) has been coming in more and more handy for me.
my favorite use of strace to learn which files (especially config files) are being open by a new daemon/tool:
strace -f -s1024 2>&1|grep open<p>also remember also useful 'ltrace' - libraries tracing
Please see<p>ftp://86.0.252.89/pub/release/website/tools/trace-20140126-x86_64-b95.tar.gz<p>This is a tool called ptrace - which does everything that strace does and a lot more. You have working binaries in there, and most of the source - I havent extricated the full build dependencies so it all builds, but this includes extra facilities like reporting summaries of process trees, showing only connections or files, and shlib injection into a target process.<p>If people are interested more on this, contact me at CrispEditor-a.t-gmail.c-o-m