Extreme debugging - a tale of microcode and an oven

I've worked at a couple of jobs (telecom, semiconductors) with environment chambers and equipment of that ilk. Heat guns were a common bench testing tool.<p>Fast forward ten years and I'm doing contract work for a local company. Of course the contractor got the shittiest workstation, which dies every afternoon like clockwork. So I move the POS white box away from the window and out of the sunlight. "Magically" it only crashes once week. So I send a ticket into the queue, stating the obvious, then the IT folks quibbled and finally admit the motherboard needs to be replaced.

Beautiful example of great engineering. I like how they, at least from how the story is told, they first formed a hypothesis based on their knowledge, found a systematic way to test/particularise it and then built a fix for production in a separate step.

The scientific method can get iterative. Observe / purpose / hypothesis / experiment / result / conclusion. It gets harder when all one's hypotheses' test cases conclude out as "Well, it's wasn't that". Or when the bug's intermittent with no apparent relation to one's domain knowledge.<p>That's the time one must apply heuristics such as (A) expand one's domain knowledge through research or talking to others, or (B) expand one's observations on the subject, or (C) expand one's hypotheses into the realm of "Oh it's just not possible that...", or "Let's try something that looks really 'stupid'...".<p>Of course, heuristic (C), being presumption-checking in face-saving disguise, has been the most fruitful heuristic in all my years. Those successes often become the best "teaching moments".<p>But here's a success story that followed heuristic (B). It goes back to the early days of the MITS Altair 8800[1] and its "S-100 bus"[2], a physically large passive backplane bus for the 8080 CPU. My employer sold complete systems comprised of that hardware with general purpose business software we wrote (using 1979-and-later versions of Bill Gates' first commercial product [3]).<p>We had systems that would intermittently "cold-crash" with no observed relationship to anything "rational".<p>So, observe: Pull the boards out and inspect them for bent pins on chips, cold solder joints... nothing. OK look at the backplane connectors, topside: No dust or bent pins. Finally turn the emptied chassis upside-down and look at the 100 solder connections for each of the 18 or so card slots. Get out a magnifying glass because this will take some time...<p>...and there they were: Three "cakey" looking broken solder joints on the bus. Evidently, the backplane board flexed enough (or the S-100 connector pins themselves ccommunicated enough flexure to their solder joints) upon board insertion and removal to break a weak solder joint on the bottom of the big PC board that was the backplane bus. Resolder all 1,800 pins and promlem solved for the lifetime of the machine. Found the same failure mode in a couple other boxes in the time 7 years I worked there, otherwise maintaining software.<p>[1] <a href="https://en.wikipedia.org/wiki/Altair_8800" rel="nofollow">https://en.wikipedia.org/wiki/Altair_8800</a> [2] <a href="https://en.wikipedia.org/wiki/IEEE-696" rel="nofollow">https://en.wikipedia.org/wiki/IEEE-696</a> [3] <a href="https://en.wikipedia.org/wiki/Altair_BASIC" rel="nofollow">https://en.wikipedia.org/wiki/Altair_BASIC</a>

This really is a fantastic article. Mayhaps you should resubmit it Monday morning or some time when it might get more traction.

I like forth. I personally have played with it by making grobot teams.<p>Creeper world will actually use a stack based language like forth for it's next release.<p>(specifically, it will use a derivative called crpl for scripting)

I enjoyed reading this story. Hunting down those kind of bugs is not something most of us will ever be able to do.

I was first exposed to Forth by a member of the PSU Timex-Sinclair UG (while I was still in high-school) and (much) later used it to program a system that controlled the terminals used by stock traders. I'm not saying I'd want to go back to writing software in Forth, but it was a strong language for control systems ... perhaps the Erlang of its day?