strace is fantastic. Since it captures the detail of (nearly) all interactions between the process and the outside world, you can use it to answer many questions.<p>Why isn't this process invocation picking up my changed lib file? (strace, see if the changed file is being opened)<p>What are the exact http req/responses being made during the problem? (strace server or proxy with large -s value to see all read/write/sendmsg/recvmsg etc)<p>This tool fails when run as user X, probably a perms problem but which file? (strace, look for EPERM failures, probably to open())<p>Which /proc files are necessary to the operation of tool X (useful when checking what will and won't run in a sandbox like dotcloud)?<p>Main restrictions that I know of(in practice only the first is sometimes a problem to me):<p>- http/ssl hides the buffer info from 'strace -s'. Another good reason for ssl offloading :-)
- IO can occur via memory reads/writes after mmap(), which strace can't see<p>ltrace is a pretty nice complement too (trace inside dynamically loaded libs).