I used to work on compilers, C/C++, COBOL, PL/I, Java, a whole bunch of them. This is actually somewhat similar to my favorite bug I encountered while implementing some stack mapping optimizations.<p>The compiler itself was crashing out while compiling a SPEC2000 test case (perlbmk I think?) with an illegal instruction. This was already quite suspect since it was branching to somewhere WAY outside of where the program usually resides, and the compiler was compiled with a compiler that's known to be nearly rock solid. I got quite lucky in that I managed to find one level of the stack trace, and it pointed me towards sprintf. Using some awesome tools some coworkers and I had developed over the years, I managed to narrow down the test case to about 5 lines of code that involved long doubles. So I grepped the compiler source code for sprintf, set breakpoints on the ones that I thought would get called, and just kept stepping through them until it finally crashed hard. Then I just reran, and stopped at the final breakpoint and started stepping through the assembly. What I saw happen just blew my mind, the code was just a simple:<p>sprintf(buffer, "fold: %Lf", result);<p>But what was happening is that the buffer was only 200 characters long, and the long double was roughly 1000 characters long. It was just a buffer overflow, that was so long it ended up overwriting the register save area, and the return address pointer. So the sprintf completed, but when it went to branch back, it loaded some characters instead of the return address. Just hilarious, and good thing I was working on stack mapping and was familiar with the stack layout of this linkage convention.<p>The solution of course was to just use snprintf instead. No sorry, that's wrong since that platform doesn't have an snprintf (yay mainframes!), and so I had to use %0.6Lg instead of %Lf.<p>Compilers are fun!
Great analysis and tenacity in hunting this one down :)<p>Letting new compilers loose on existing codebases is always fun and you learn lots of things in the process, I can only recommend it. I debugged a problem once that also had to do with interfacing runtime-generated code with compiletime-generated code. There were differences in the expectations of the ABI, which is described in this bug:<p><a href="http://llvm.org/bugs/show_bug.cgi?id=12207" rel="nofollow">http://llvm.org/bugs/show_bug.cgi?id=12207</a><p>It only surfaced when compiling the codebase with clang (previously gcc). Took quite some digging to find the problem.
Thanks for sharing -- I love puzzles like that.<p>Just curious: Were the lldb session snippets taken from the original debugger session? I keep getting weird looks when I use a command-line debugger just to have a transcript afterwards. (The weird looks being from people who'd rather send me a screenshot of their GUI debugger's call stack.)
Haha, it's almost like some detective short story.<p>Surprisingly it wasn't as hard to follow as I thought. Maybe I'm starting to get good at this Computer Science thing.