Debugging an evil Go runtime bug

Marcan's attitude is great; I know of a ton of people (myself included) who would've written that article with far more complaining interleaved. Super informative as well, I learned a ton from this article (GRUB 2 feature for marking off bad RAM? Wow!). Very well written, informative, humorous, etc. Love it

That is such as well written post, and showcases some serious diagnostic skills by an experienced person.<p>Btw, thanks about telling me about the badram Grub 2 feature, had no idea that exited.

Man I loved the approach to narrow down the offending object file using the regex on the SHA hash of the binaries. That would&#x27;ve saved me lots of time hunting and guessing bugs with cscope+kdb back in my kernel hacker days!

Wow, these are some serious debugging skills. I also admire the tenacity and the will to investigate the root cause.<p>&gt; I tried setting GOMAXPROCS=1, which tells Go to only use a single OS-level thread to run Go code. This also stopped the crashes, again pointing strongly to a concurrency issue.<p>I think I would have stopped there.

Hah. I remember Bryan Cantrill complaining about this exact thing. Glad that it&#x27;s fixed.<p>Turns out somebody else did, too: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;bcantrill&#x2F;status&#x2F;774290166164754433?lang=en" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;bcantrill&#x2F;status&#x2F;774290166164754433?lang...</a><p>&#x2F;edit: spelling

The investigation in the linked Go issue [1] is also impressive.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;golang&#x2F;go&#x2F;issues&#x2F;20427" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;golang&#x2F;go&#x2F;issues&#x2F;20427</a>

This is great story, probably wins the year for &quot;best bug you&#x27;ve ever encountered?&quot; question. Having implemented some weird runtimes for weird languages, I am sympathetic to Go team here -- these odd tradeoffs of pushing the envelope on OS &lt;-&gt; your_own_compiler interactions can trigger some wild experiences.

&gt;Since the problem gets worse with temperature, what happens if I heat up the RAM?<p>Neat. I wonder if that makes Rowhammer more likely to occur.

For a similar tale of vDSO getting someone in trouble, check out this fun talk &quot;Really crazy container troubleshooting stories&quot;: <a href="https:&#x2F;&#x2F;media.ccc.de&#x2F;v&#x2F;ASG2017-115-really_crazy_container_troubleshooting_stories" rel="nofollow">https:&#x2F;&#x2F;media.ccc.de&#x2F;v&#x2F;ASG2017-115-really_crazy_container_tr...</a>

Ninja level debugging and diagnostic skills. A fascinating read from start to finish. Bonus points for the GRUB 2 feature for masking out bad RAM blocks – still dreaming of owning a laptop with ECC memory :&#x2F;

Setting up for a 104 byte stack seems pretty crazy, wouldn&#x27;t you risk overrunning the red zone even without all that stack probing? <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Red_zone_(computing)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Red_zone_(computing)</a>

Such a thorough and well written write up.<p>To think that some experienced programmers I know declare that concurrency is easy.

I learned so much from that post. The author clearly love tinkering with computers. I wish I had that same leveled curiosity. Well done.

this is incredibly impressive

Why doesn&#x27;t the vDSO code just use MOV in its stack probe probe rather than an OR?

Thanks for taking the time to do this writeup! Super fun to read and informative.

Not only he does that, he also explains the debug procedure like we&#x27;re 5 years old. I&#x27;m so impressed.

I wish i can do something like this one day :) Impressive skills by the author! Well done!

An awesome mystery story!