Worst in what sense? Did the most damage, or hardest to find, or just most mind-blowing?<p>We had a customer whose Unix environment didn't pass a stderr file handle to running programs. So if you tried to print something to stderr, it scribbled on whatever file happened to be open on handle number 2. This got reported to us as a bug. After looking at it, we told them to fix their insane runtime environment.
I've probably seen worst than this but this one comes to mind.<p>A sysadmin had (I can only assume accidentally) removed the SUID bit from the crontab binary causing an application we used that invoked the binary to not work. To even allow this to work we had a custom SELinux module that we suspected had the issue.<p>I ended up comparing systrace output between a working instance and the broken instance (in prod) to track it down.
Most mind boggling: a three character long comment addition in java code broke the build pipeline. It was 100% reproducable. Nothing else was in the commit. Reverting fixed it, re commiting broke the build again.<p>It happened because the mvn on the build server ran out of memory exactly because of that 3 character addition.
I can't think of a particular bug, but I encountered one database so horribly misused, so poorly configured, for which no staff was given proper training, and with no data sanity checks in place, such that the university department using it had lost in excess of $700K revenue through one of its services over the course of 3 years.