I've had enough years to become wiser, become a fanatic for configuration management, and get over the embarrassment: I'm the consultant that screwed things up. Some background: the Stat department was running a variety of systems besides the Solaris workstations, and there was, within UNC-CH, a separate support organization that was cheaper and more comfortable with Microsoft products where Stat was sending their support dollars. When that organization needed Unix support, they called my employer, Network Computing Solutions, and I showed up.<p>There was effectively no firewall at UNC-CH at the time (something something academic freedom something something), and the Stat Solaris machines were not being regularly patched. Uninvited guests had infested them, and it appeared the most likely entry point was sendmail - at the time, it was the most notorious vulnerability on the internet. Since my preference to wipe and reload was unacceptable - too much downtime and too many billable hours - the obvious thing to do was update sendmail. The rest is history.
Source:<p><a href="https://www.ibiblio.org/harris/500milemail.html" rel="nofollow">https://www.ibiblio.org/harris/500milemail.html</a><p>FAQ about the story:<p><a href="https://www.ibiblio.org/harris/500milemail-faq.html" rel="nofollow">https://www.ibiblio.org/harris/500milemail-faq.html</a><p>Past submissions:<p><a href="https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=500%20mile%20mail&sort=byPopularity&type=story" rel="nofollow">https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...</a><p><a href="https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=http%3A%2F%2Fweb.mit.edu%2Fjemorris%2Fhumor%2F500-miles&sort=byPopularity&type=story" rel="nofollow">https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...</a><p>(It seems I can’t find a search query which includes both urls)
I love that this took a perfect storm of having a statistician and sys admin both bent on finding the cause of a weird intermittent problem in their own way.<p>This could have happened a million times where the story was a lot less interesting:<p>"Hey, I'm having weird intermittent problems sending email."<p>"Hmm, we're using the wrong version of Sendmail. All fixed, case closed."
Best part of reading this is coming away having learned the existence of units the CLI. How did I spend 20 years on the shell and not have needed or discovered this?
I used to collect these kind of stories:<p>When I flush my toilet my computer reboots:
<a href="http://www.techtales.com/tftechs.php?m=199712#66" rel="nofollow">http://www.techtales.com/tftechs.php?m=199712#66</a> (the first story on the page)<p>If I buy vanilla ice-cream my car wont start:
<a href="http://www.netscrap.com/netscrap_detail.cfm?scrap_id=501" rel="nofollow">http://www.netscrap.com/netscrap_detail.cfm?scrap_id=501</a><p>A specific cargo routing crashes system:
<a href="https://www.jakepoz.com/debugging-behind-the-iron-curtain/" rel="nofollow">https://www.jakepoz.com/debugging-behind-the-iron-curtain/</a><p>Tape-drive failure only within large print jobs:
<a href="http://patrickthomson.tumblr.com/post/2499755681/the-best-debugging-story-ive-ever-heard" rel="nofollow">http://patrickthomson.tumblr.com/post/2499755681/the-best-de...</a><p>Interplanetary debugging with the Mars Rover:
<a href="https://www.eetimes.com/the-trouble-with-rover-is-revealed/#" rel="nofollow">https://www.eetimes.com/the-trouble-with-rover-is-revealed/#</a>
The ending of this makes it sound super clean. 3 ms * speed of light => ~560 miles. "It all makes sense!"<p>But ... isn't the speed of light through fiber actually like 2/3 of the speed of light in a vacuum? And that fiber isn't going to be laid out in a straight line exactly to the destination. So I think really there must have been a fair bit of uncertainty around that ~3ms to abort a connection.
Truly this is a worldly gem. Thank you for submitting this. :)<p>It's easy to forget that, even though transmissions still travel at near lightspeed, it still takes more than an instant to reach its destination, even digitally. I should keep this in mind, I think.
IMO even though this has been posted a bunch of times it’s important to repost these sort of campfire ghost stories so future developers can think more creatively about strange errors.
Only the other week we were doing some testing on a new HCI (hyper-converged infrastructure) I'm doing the network for. At the end of the test period, we were having some storage sync issues. Everything seemed to PING ok, except my colleague happened to notice large jumbo frames over 8000 bytes were getting dropped. I double checked that we hadn't inadvertently changed network configuration. It was only by chance we had another test looking at transceiver signal levels that a customer engineer saw an alarm on RX level. It was then we remembered one test was to remove a module. I then noticed some error counts. We shutdown that particular link until we could visit the site. Sure enough, that fibre wasn't quite clicked in anymore. There was enough of a bridge across the fibre air gap for shall packets, but just wide enough so large packets statically couldn't be corrected enough to work.
A while back, Andreas Zwinkau collected this kind of stories.
Please post him if there is anything not yet listed.<p><a href="http://beza1e1.tuxen.de/lore/index.html" rel="nofollow">http://beza1e1.tuxen.de/lore/index.html</a>
If somebody is tracking apt/yum downloads of packages, they might see a sudden spike for "units". Just installed and it is a nifty little useful tool.
Love this story.<p>I'm certain it will continue to be reposted to this website until ceases operations or the heat death of the universe whichever comes first.
Can someone please explain to me the POP reference? I do not understand what this author means by that.<p>I also would like help understanding what $ units gives? The command looks to be "units", but where do the numbers he entered in come from? I would appreciate this extra context.
Edit: Found it - its /usr/share/units/definitions.units (on Pop!_OS, so probably same on Ubuntu/Debian).<p>The FAQ[0] mentions:<p>> units on SunOS doesn't know about "millilightseconds."<p>> Yes. So? I used to populate my units.dat file with tons of extra prefixes and units. And actually, I think I was using AIX to run units; I don't know if it knew about millilightseconds. Take a look at the units.dat shipped with Linux these days. It definitely knows about millilightseconds.<p>I tried locate for units.dat but couldn't find it. Anyone knows where is it? Not keen on running a system-wide find.<p>[0]: <a href="https://www.ibiblio.org/harris/500milemail-faq.html" rel="nofollow">https://www.ibiblio.org/harris/500milemail-faq.html</a>
Hat tip to @nfriedly from back in 2011:<p>FYI units on OS X doesn't recognize millilightseconds, but you can do this:<p><pre><code> You have: 3 lightyears / 365 / 24 / 60 / 60 / 1000
You want: miles
* 559.21802
/ 0.0017882113</code></pre>
This story makes me smile every time it comes up. It's fascinating how many arbitrarily coded limits we keep breaking as we make our tech go faster without re-assessing the original assumptions :)
Haha, this was interesting. I posted the same story with more or less the same title a couple of month ago [1], and no one saw it and no comments. This time it got almost 1000 points and a lots of comments.
What I think is interesting is how the same thing can get so different traction. Wonder what factors it is that makes a thing get traction and not?<p>[1]
<a href="https://news.ycombinator.com/item?id=22164691" rel="nofollow">https://news.ycombinator.com/item?id=22164691</a>
Since I've worked with linux email servers (sendmail, qmail, postfix, exim, etc) practically my whole profissional life (since around 1997, but used BBS since 91 - I'm 41 now), this story really amused me and got my attention! I <i>love</i> this kind of email debugging! LOL
There is a popular GitHub repository for similar debugging stories: <a href="https://github.com/danluu/debugging-stories" rel="nofollow">https://github.com/danluu/debugging-stories</a>
Soundtrack for this story: <a href="https://www.youtube.com/watch?v=tbNlMtqrYS0" rel="nofollow">https://www.youtube.com/watch?v=tbNlMtqrYS0</a>
I've heard this story before, but i didn't realize it was as recent as 2002. It feels like something from a much earlier bygone era, like the early 90s
The one thing that throws a wrench in this story for me: Lattes.<p>Latte's in 1994? In North Carolina? No way. Maybe on the West Coast, but I moved to Cali in 1989 and they were a rarity until the mid-late 90's. There were only 425 starbucks in the US in 1994 (from their site). The "fancy coffee" craze was just a blip on the radar in the mid 90's but gaining momentum.<p>;-)