Embedded systems often have crappy compilers. And you sometimes have to pay crazy money to be abused, as well.<p>Years ago, we were building an embedded vehicle tracker for commercial vehicles. The hardware used an ARM7 CPU, GPS, and GPRS modem, running uClinux.<p>We ran into a tricky bug in the initial application startup process. The program that read from the GPS and sent location updates to the network was failing. When it did, the console stopped working, so we could not see what was happening. Writing to a log file gave the same results.<p>For regular programmers, if your machine won't boot up, you are having a bad day. For embedded developers, that's just a typical Tuesday, and your only debugging option may be staring at the code and thinking hard.<p>This board had no Ethernet and only two serial ports, one for the console and one hard-wired for the GPS. The ROM was almost full (it had a whopping 2 MB of flash, 1 MB for the Linux kernel, 750 KB for apps, and 250 KB for storage). The lack of MMU meant no shared libraries, so every binary was statically linked and huge. We couldn't install much else to help us.<p>A colleague came up with the idea of running gdb (the text mode debugger) over the cellular network. It took multiple tries due to packet loss and high latency, but suddenly, we got a stack backtrace. It turned out `printf()` was failing when it tried to print the latitude and longitude from the GPS, a floating point number.<p>A few hours of debugging and scouring five-year-old mailing list posts turned up a patch to GCC (never applied), which fixed a bug on the ARM7 that affected uclibc.<p>This made me think of how the folks who make the space probes debug their problems. If you can't be an astronaut, at least you can be a programmer, right? :-)
I’ve spent 30 years working on compilers.<p>They have bugs. Lots of them.<p>With that in mind, the article is correct that the vast majority of issues people think might be a compiler bug are in fact user errors and misunderstanding.<p>My experience actually working with users has been somewhat humorous in the past, including multiple instances of people completely freaking out when they report something that turns out to be a miscompile. I’ve seen people completely freaking out, to the point that they no longer felt that any code could be trusted since it could have been miscompiled in some way.
Back when I worked on the MPC-HC project we found a bug in the Visual Studio MSVC compiler. When we upgraded from VS2010 to VS2012 subtitles would fail to render.<p>We eventually traced it down to a small for loop that added 0.5 to double members in an anonymous struct. For some reason these three factors: an anonymous struct, double datatypes and a for loop caused those member variables to become uninitialized.<p>We extracted this code into a small code sample to make it easily reproducible and reported it to Microsoft. Their compiler team called it one of the most helpful reports they'd gotten and confirmed it was a bug in their for-loop vectorization code. The compiler appeared to have messed up the SIMD instructions to write the results of the addition back to memory.
There's 830 open and confirmed wrong-code bugs in GCC at the time of writing. Compiler bugs aren't as rare as people think: <a href="https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=SUSPENDED&bug_status=WAITING&bug_status=REOPENED&cf_known_to_fail_type=allwords&cf_known_to_work_type=allwords&f1=keywords&o1=substring&query_format=advanced&v1=wrong-code" rel="nofollow">https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=NEW&bug_...</a><p>I think it's just common for people to assume they're wrong and change things blindly rather than carefully checking the standard for their language (assuming their language even has a standard to check). It doesn't help that before AddressSanitizer and co. existed compilers would just do all sorts of nonsense when they detected possibly undefined code in C and C++.
Oh man. I uncovered a hash implementation bug in go, ca 2014 or so and I spent like two days prepping my bug report, tests, I was so certain it was me. The team of course was super nice and like ‘good catch’. Victory lap day for any nerd.
Learning to code (C) I thought I found a compiler bug lots of times and was almost always wrong. It gave me the heuristic that if I thought I found a compiler bug, it was time to take a break, have a snack and go for a walk or something before looking again. It usually helped me find my mistake much faster.<p>The thing I disliked most about later learning PHP or Javascript was that my previously usually wrong reaction of "the compiler is insane" suddenly turned out to be commonly true. Even when it wasn't an actual bug PHP and javascript were often so poorly designed that intended behaviour wasn't much better than one.
Similarly, I once ran into a broken implementation of a Dictionary type (in Mono, I think.) It was only comparing the keys' hash codes, not the keys themselves. In most scenarios this turned out to be more than good enough - for int32 keys obviously it will work, and for most strings it works too if the hash function is good - but I had a great many keys without an amazing hash function for them.<p>It's funny how sometimes a really glaring bug can hide in a stdlib for months or years just because by luck the stars never align to trigger it where somebody can notice it. In my case, the dictionary bug was causing recoverable errors, and I only noticed because I dug in instead of going "Mono's just broken".
It depends really which compiler you are testing and whether the version you are testing has just been released or has been around for some time. If the compiler is for a niche language, then it's possible to find bugs. If the compiler has been released, it's even possible to be the first person to note the bug. But the bigger the language, the more has passed, the less likely this is.
Note... it's not really that "it's never a compiler bug," but more like "it's never a backend/codegen bug."<p>It's not particularly hard (for someone who knows the language rules, which <i>are</i> difficult for a language like C++) to make a widely-used compiler be erroneous in its acceptance or rejection of code.<p>What's <i>much</i> more difficult ("never" happens) is to make the compiler accept valid code and then generate an incorrect executable. It's possible (and I run into this maybe once a year doing unusual things) but it's really rare. If you think <i>that's</i> what's going on, it's very unlikely to be the case.
I hit a similar issue in 2017 which is still the case today: Python's builtin `random.shuffle` destroys numpy arrays passed into it [0]. This is apparently a design limitation within numpy and cannot be detected or fixed, so it still stands today. I spent hours combing through my own code wondering where the bug was, because there was no way that it was caused by numpy or Python, but eventually all the likely scenarios got ruled out...<p>[0] <a href="https://github.com/numpy/numpy/issues/10215">https://github.com/numpy/numpy/issues/10215</a>
My worst slowdown ever was when a compiler failed because a bit had flipped somehow. After a month and a half I finally reinstalled it and everything worked perfectly.
I still have a recognition letter from Borland regarding a bug I have found in Turbo Pascal 6.0.<p><pre><code> function BrokenResult: Integer;
var
BrokenResult: Integer; (* This should not happen *)
begin
BrokenResult := 42 (* Local variable will be assigned, function result is whatever the compiler comes up with*)
end;</code></pre>
At my first job, it actually was a compiler error, and I'm not sure if my manager ever believed me. We were using an internal gcc fork and cross-compiling, so who knows where the bug was, but the compiler team got back to me. Jump tables were sometimes broken, and we had to add a switch to disable them.<p>Not the right lesson to learn for a first job.
As with many others commenting here, I've certainly had many suspicions I'd found a compiler (or equivalent) bug <i>almost</i> always proved false.<p>But in 30+ years of professional experience, I've also found two compiler-like bugs (I tend to use scripting / interpreted languages, so "compiler" isn't entirely accurate). One was in a commercial software package in which the documentation and implementation of a feature were reversed (what resolved as "true" should have been "false" and vice versa). That resulted in a code fix.<p>And another was a bug (specifics of which I've since forgotten) in GNU Awk, and not, I painstakingly verified, in my own code. That was also submitted and fixed.<p>Every other time, though, my own damned fault ;-)
Discussed at the time:<p><i>“It is never a compiler error”</i> - <a href="https://news.ycombinator.com/item?id=15699675">https://news.ycombinator.com/item?id=15699675</a> - Nov 2017 (272 comments)
I crashed the Oracle HotSpot Java virtual machine back in 2017 with a totally innocuous program involving nested arrays. After reproducing and minimizing it, I filed a bug report. It got fixed quickly.<p>I'm not sure why the page is no longer publicly available: <a href="https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8181921" rel="nofollow">https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-818...</a> (JDK-8181921)
Our infrastructural team keeps about 2 MSLOC building on several compilers and running on several architectures. They report a new compiler bug every 2-3 years.
Around 10 years ago I found a JIT bug in a JDK from a big Java vendor. A new version of a web application server had been applied. The production application crashed after around 30 minutes of running, almost simultaneously on both production sites. It was an internal checksum calculation in the application that failed - an obscure error never seen before. The upgrade was rolled back immediately. I was assigned to the case and of course didn't suspect a JIT error. But within a week of investigation I started suspecting it must be (but I didn't dare tell anyone!) and I eventually managed to show this and reproduce it consistently. The vendor confirmed and made a temporary workaround via switches that disabled some new optimizations. Later a real fix was shipped.<p>I've also found 3-4 JavaScript JIT compiler errors in major browsers, all confirmed. I was a developer on what was for its time a quite complicated JavaScript solution, so we tended to encounter obscure JavaScript errors before others.
In my case, it wasn't a compiler bug - it was a bug in the STL, before the STL was part of the compiler. It was a separate thing you downloaded. I found a bug, and emailed Stepanov (or Lee - I forget). Me, just some random nobody on the internet. I got a fix, and then an improved fix, and then a final fix, all within two hours. I was <i>floored</i>.
While it's <i>almost</i> never a compiler error, it happens, and I have personal experience; I once found an error in the VAX/VMS Pascal compiler - and could demonstrate it as such by disassembling the compiler output - and had to work around it until DEC fixed it.
This brings back memories of XL calculating an address wrong as a result of it lying on a boundary ≡ 0 (mod 2^32). Fortunately, the TOBEY (XL back-end) guys were in the same area in the building so restablishing our sanity was faster than it otherwise could have been...
In the early days of C++11, I used to get unique ICEs in both GCC and Clang weekly. One particular annoyance was when a stable release of Debian decided to ship a point release with a <i>regression</i> (not looking it up, but it was something like: 4.6.1 or 4.6.3 worked, but 4.6.2 had completely broken UDLs for constant expressions or something). I had <i>just</i> converted the whole codebase to use UDLs aggressively since they worked everywhere in my tests, not thinking I had to test every point release in between ...<p>Thankfully I don't think I ever had any miscompilations - that would require the code actually compile across several compiler versions in the first place.
As a compiler developer, I see plenty of bugs. So, it's sometimes a bug. But, in the case of C (and C++ by extension), it's often a language design bug that unfortunately has no fix and can only be worked around.
The article is right: it is almost never a compiler bug. I have had that experience of reporting and being wrong. It sucks.<p>On the other hand, I have a confirmed bug in Clang [1] and a non-rejected bug in GCC [2], so it does happen.<p>[1]: <a href="https://github.com/llvm/llvm-project/issues/61133">https://github.com/llvm/llvm-project/issues/61133</a><p>[2]: <a href="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108448" rel="nofollow">https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108448</a>
Related "It Is Never a Compiler Bug Until It Is" <a href="https://news.ycombinator.com/item?id=24636326">https://news.ycombinator.com/item?id=24636326</a>
>> It is not a compiler error. It is never a compiler error (2017)<p>No, not <i>always</i> true. Even in modern compilers -- as matured and as modern as VS 2022-- you would still get bug.<p>I found one[0]. In my case it's easy to tell it's a compiler bug because the program just can't compile properly. But it's also not easy to reproduce, which just proves how well tested compilers usually are.<p>0: <a href="https://github.com/dotnet/roslyn/issues/74872">https://github.com/dotnet/roslyn/issues/74872</a>
I wonder if the bubble-sort implementation in this library helped prolong the life of this bug. Most people would choose another impl for performance reasons, and thus not find this bug.
I was playing with Java 1.0.1 trying to make an app screen with a GridBagLayout. It made utter hash of my layout, drawing things on top of each other, etc. Applying the First Rule of Compiler/Runtime Bugs I double-checked and triple-checked and quadruple-checked my work, making sure I used the GridBagLayout API exactly according to spec. Eventually I posted to USENET comp.lang.java asking, "Is there a bug in GridBagLayout?"<p>The problem disappeared in Java 1.0.3.
Back when I was using CodeWarrior to make a game for PlayStation 2, I found a compiler bug, but fortunately, it was one where it gave an error on valid code, rather than generating bad output. I can't remember the details, but I had some sort of equation that my co-workers agreed should have compiled with no problems. I was able to rewrite it a little to get the result I wanted without triggering any compiler errors.
Just last week I tripped over a couple compilation bugs in (an old version of) bpftrace.<p>One was caught by internal checks somewhere, something about struct member offsets that I think was an alignment / padding issue and didn't seem to actually break anything. The other made it segfault during compilation, and I had to just tweak my code blindly until it decided to go away.
When I started learning Turbo Pascal I came across a problem where an if-statement was obviously decided wrong. I saw the values in the debugger.<p>My rescue was that I had a more experienced friend who knew that IIRC the compiler would choose the data type of the left operand of a comparison also for the right operand leading to potential sign switches.
I’ve hit so many fun compiler bugs. Usually easy to work around though (yay modern / fp flavored languages). It certainly helps when it also crashes the compiler ;).<p>Miscompilation bugs are definitely nasty though. Especially if it’s a self boot strapping compiler. Save your old build artifacts! :)
As the article shows it’s highly dependent on which compiler you’re relying on. Always good to keep this in mind when assessing the likelyhood of an error.
I’ve thought I’d found a compiler but maybe 5 times in my life and it has never actually been a compiler bug.<p>When I reflect on the ~25 years I’ve been programming C, all of the times I thought I’d found a compiler bug were in the first ~8 years. Dunning-Kruger hard at work :-/
No, it is very often a compiler bug.
Just look at the gcc, clang or rustc tickets.<p>e.g. <a href="https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=__open__&no_redirect=1&order=Importance&product=gcc&query_format=specific" rel="nofollow">https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=__open__...</a><p>It's massive, and several gcc versions have to be blacklisted. The clang restrict bug is still not fixed, it never worked.
rustc was never memory-, type- nor concurrency-safe.
...unless it is. Compiler crashes are easy to see, but it can actually be nontrivial to identify miscompilations as they can only trigger in certain code paths and with careful observation you can notice the second order effects...<p>If you specifically look for them you might find quite a bit: <a href="https://web.cs.ucdavis.edu/~su/publications/emi.pdf" rel="nofollow">https://web.cs.ucdavis.edu/~su/publications/emi.pdf</a> [disclosure: an author]