“Unexplainable” core dump (2011)

&gt; Our code and compilers are constantly changing, and the problem disappeared as suddenly as it appeared ... only to happen again 2 years later in a completely unrelated executable.<p>It does not encourage me how much this sounds like the short story &quot;Coding Machines&quot;. The original post even happened right about 2 years after the short story was posted, then in that comment reoccured after another 2 years.<p><a href="https:&#x2F;&#x2F;www.teamten.com&#x2F;lawrence&#x2F;writings&#x2F;coding-machines&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.teamten.com&#x2F;lawrence&#x2F;writings&#x2F;coding-machines&#x2F;</a>

One of the best bugs I&#x27;ve seen had a description fairly similar to this. Hot routine run at scale (floating point math for ads ML training) fails at a rate about 0.000000001. Turned out to be a very obscure bug in the context switching code in the linux kernel, the FP registers weren&#x27;t being restored properly.<p>Another one, the debugging was aided by the fact the developers ensured that everything was accessed through const pointers, so it wasn&#x27;t their code corrupting their memory.

In this CPU, AMD first introduced the &quot;stack engine&quot;. This is likely a bug in that feature. It has a speculative stack address delta register in the front-end that is updated directly with push&#x2F;pop instructions, and that delta is dispatched with the stack memory uop to be added to the original stack address register when doing address generation in the load&#x2F;store units.<p>The delta has to be small, because you don&#x27;t want big adders in the front end and because the delta has to be sent down with the push&#x2F;pop memory uops. That means it can overflow or underflow, at which point it has to be reset by sending a synchronize operation to the back-end to update the original stack register (agner has a better description).<p>So the delta register is probably 10-12 bits on Barcelona, and this bug is probably a corner case where the stack register update is happening, hence 1024 bytes off. Perhaps there is a window where a uop can get the old delta value, but the new base value (or vice versa) when a sync is operation is concurrent.<p>Setting that MSR value possibly disables that stack engine feature entirely, or it could be it disables some aggressive and complicated detail where the bug is, e.g., allowing stack operations to run concurrently while flush operations are in progress.<p>It&#x27;s not a coincidence there just happens to exist a way to disable this at runtime. The way processors are designed means that everything must be able to be observed, debugged, <i>and fixed</i> in the field. That means everything has to have fine-grained ability to control, disable, enable safer fallback paths, and even engage additional logic to reduce the state space in some cases (e.g., serialize pipeline while a particular operation occurs).<p>Usually these bug fixes decrease performance (except in cases where a performance bug is found and the fix actually increases performance), so you want the switches to be very fine-grained. So it&#x27;s possible they fixed the stack engine bug without disabling it entirely.<p>It would be like shipping software and providing support and bug fixes for it for the next 5-10 years without patching the software, only updating the config file. It&#x27;s quite amazing. For every one of these issues that hits the field, there will be many found internally during the internal hardware bring up and verification (which will be ongoing for at least part of the life of the CPU).

Reminds me of what we dubbed &quot;the cosmic ray incident.&quot;<p>During college, we were arranged in groups of 2 or 3 to do some pair programming for the more complicated exercises.<p>I had attached the debugger and had set a variable to 99, so the loop would execute one more time and we could test if our changes would work.<p>Went through the instructions step by step, and suddenly we got a segmentation fault. Between setting it to 99 and it accessing some data later on, the value changed to 107.<p>Quite a bit of confusion ensued. I made a backup of the executable before recompiling. Running them again, both the new version and the backup worked perfectly. The files matched bit for bit. To this day we have no idea what caused that bit flip.

Another good source of &quot;impossible&quot; bugs is overclocking.<p><a href="https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;oldnewthing&#x2F;20050412-47&#x2F;?p=35923" rel="nofollow">https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;oldnewthing&#x2F;20050412-47&#x2F;?p=35...</a>

One of the best bugs I&#x27;ve seen had a description fairly similar to this. Hot routine run at scale (floating point math for ads ML training) fails at a rate about 0.000000001. Turned out to be a very obscure bug in the context switching code in the linux kernel, the FP registers weren&#x27;t being restored properly.

I had something similar to some of the things described here, with a dumber cause. Started getting hardfaults in my firmware after a context switch. Thought bad array access, incorrect FPU settings, misbehaving interrupt. All dead ends.<p>Realized that it was only happening when a particular motor was running, then further isolated to when it was only running at higher duty cycles. I had tested the motor in isolation but at low speeds, so I didn&#x27;t see issues until I tried to run the full application. Turns out the EE had royally screwed up the current sense circuit, and the MCU on the ADC was seeing voltages lower than -1V, well beyond the absolute max ratings for the chip.<p>I guess ST doesn&#x27;t make guarantees about what happens when you violate those ratings, but a corrupt stack pointer is not what I would have expected.

This is great. Reminds me of a crash I saw early in my career. It was a null-pointer exception, except it occurred right after confirming the address was non-null. This was on a single core with a non-preemptible kernel. So the processor just took the wrong branch! There was simply no other explanation.

I recognize this StackOverflow user name. I have read many of their answers. They are usually <i>excellent</i>.<p>See more here: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;50617&#x2F;employed-russian" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;users&#x2F;50617&#x2F;employed-russian</a>

This reminded me a problem that I have seen almost 17-18 years ago. There was a file that we were trying to download but it was just being interrupted during download process. When we tried to download the file on another computer, it was fine. Long story short; we changed the ethernet card of that computer and it started downloading file. I don&#x27;t remember all the details; but probably something wrong witj ethernet card driver.

I think the closest I ever came to this was when I stumbled into a bug in git.<p>We were asking git to do something utterly trivial, like clone, and it was segfaulting. We installed the debug symbols or something (as I recall you can install these separately in Debian) and started trying to see where the function was crashing. The crash was in a parser, and the code was doing,<p><pre><code> char *foo = strstr(input, &quot;something constant&quot;); </code></pre> and segfaulting later (but not chasing null!), during the first use of foo. I figured that input must, therefore, be a bad pointer, since the string literal is by definition fine, and the only way to get a bad pointer <i>out</i> of strstr was to give it bad input in the first place.<p>So, in gdb, I print the input point. It&#x27;s valid. &quot;Damn&quot; I think to myself &quot;we&#x27;ve gotten unlucky, and whatever triggers the cascade of UB hasn&#x27;t happened this run. Freakin&#x27; heisenbug.&quot; So, I told gdb to just continue running the program: it crashed. Same backtrace: dereferencing foo triggered a segfault, and not with a null pointer.<p>If the input pointer to strstr is valid, how can the output be a bad pointer? strstr is documented as,<p><pre><code> These functions return a pointer to the beginning of the located substring, or NULL if the substring is not found. </code></pre> So, what gives?<p>I attempted to debug strstr … that was almost a mistake. strstr is, unfortunately heavily optimized. It might have been this one: <a href="https:&#x2F;&#x2F;sourceware.org&#x2F;git&#x2F;?p=glibc.git;a=blob;f=sysdeps&#x2F;x86_64&#x2F;multiarch&#x2F;strstr-sse2-unaligned.S;h=c6aa8f45a60d175826c3d4e6eb483495a2cbf424;hb=f2698954ff9c2f9626d4bcb5a30eb5729714e0b0" rel="nofollow">https:&#x2F;&#x2F;sourceware.org&#x2F;git&#x2F;?p=glibc.git;a=blob;f=sysdeps&#x2F;x86...</a><p>I&#x27;m okay with a debugger, but I&#x27;m not going the be able to follow where vectorized code is going wrong. Almost stupidly, I told strstr to (f)inish the function&#x27;s execution, at which point gdb printed the return value: it was a valid pointer!<p>&quot;Odd. Is this run going to succeed?&quot; I continue execution: <i>it crashes</i>.<p>I restart the debugging, and run to the completion of strstr: the pointer is correct, but again, crash. <i>How.</i> The segfaulting address isn&#x27;t the pointer strstr is returning, either, and nothing modifies foo after the strstr.<p>I single step out of strstr, and immediately print foo. The resulting pointer is wrong. strstr is working fine … but what, the assignment operator is broken? At this point, I&#x27;m sure I&#x27;m crazy. I disassemble the source, and lo and behold, the disassembly is trivially wrong. Forgive me as I can&#x27;t amd64 unless I&#x27;m looking at it, but the disassembly is something like,<p><pre><code> call strstr cltq &lt;store foo&gt; </code></pre> We look the odd instruction up; it sign extends eax into rax. That&#x27;s … not a valid operation on a supposed char pointer … there&#x27;s not even technically a value in eax at this point. strstr returned a pointer in rax. eax is just the lower 32 bits of that pointer, and sign extending that back into rax makes no sense. It&#x27;s like the compiler thought the return from strstr was a signed int … and that&#x27;s where it hits me: C is shooting us in the foot again.<p>&gt; <i>and if no declaration is visible for this identifier, the identifier is implicitly declared exactly as if. in the innermost block containing the function call. the declaration</i><p>&gt; <i>extern int identifier();</i><p>&gt; <i>appeared.</i><p>The default return for an undeclared function, in C89, is int. (I&#x27;m actually not sure what C11 says about this. The C89 rule appears to be gone, but there doesn&#x27;t seem to be anything in its place, which is bizarre. &quot;Undeclared&quot; does not appear, except for an unrelated footnote.)<p>Slap in the proper include for strstr, recompile. No segfault, disassembly is correct.<p>Upgrade to latest version, disassemble: disassembly is correct. Someone else fixed the bug, in the meantime. Should have just tried upgrading in the first place…<p>(After typing this up: had to dig up the debugging session. It was strchr, not strstr. Potato, potato. Someday … this should be a blog post…)

Sounds fun, wish I had the talent to do some serious debugging.

If you are lucky, you&#x27;ll debug a CPU bug once in a lifetime.<p>If you&#x27;re really lucky, you&#x27;ll never see one at all!

Debugging that must&#x27;ve been a PITA for sure.