Troubleshooting is one of my main comparative advantages - I'm better at it than I am at programming and I enjoy it more. It's also a relatively independent skill, not everyone is good at it or likes it. It reminds me of the lateral thinking puzzles I did as a kid where you had to ask questions to uncover whatever the weird situation was. You have to question your assumptions - think about how you might be wrong, something I like to do in general anyway.<p>There's a certain way of reasoning about the problem and thinking about what it might be with limited information in a systemic way. It's also a bit broader than debugging - you can do a lot of troubleshooting (sometimes faster and more effectively) by doing things other than reading the code.<p>It's also been somewhat of a career advantage because it seems to be both more uncommon than standard dev for someone to be really good at and something that most people dislike (while it's my favorite thing to do). It also overlaps a lot with other more general types of problem solving.<p>Anyway - a lot of the article resonates with how I think about it too.
> Realizing that I spend more time troubleshooting than I do building or doing ...<p>That's not good. The problem with troubleshooting is that it messes up with your reward system. After you fix a hard-to-debug problem, you feel a sense of accomplishment. Which would be ok, but the problem is that this sense of accomplishment is often time higher than it should be. You go home at the end of the day thinking "well, today I didn't build anything, but it's fine, because I fixed that bug". You are becoming complacent.<p>If you end up saying to yourself, like the author of this blog here, that you troubleshoot more than you build or you do, then you have a problem. Soon you'll be seen by others as a car mechanic. Maybe a reliable car mechanic. But reliable car mechanics don't get paid a lot.<p>This might be a controversial take but here it is: being proud of your troubleshooting skills sits somewhere between being proud of your typing speed and being proud of your word document formatting skills. These things never go obsolete, but don't fool yourself into thinking they are gold currency on the job market.
I feel like the article is just a very long-winded way to say what I try to help junior devs understand and that's: take a step back and start from the very top, change only one thing at a time, don't become fixated.<p>So many issues turn out to be the smallest configuration mishaps, this is why I also promote using a debugger as much as I can as well - there's nothing quite like being able to see _exactly_ what's going on.
From zen and the art of moto: ‘There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’
Hugged to death for me right now, here's an archive link: <a href="https://web.archive.org/web/20250228192142/https://www.autodidacts.io/troubleshooting/" rel="nofollow">https://web.archive.org/web/20250228192142/https://www.autod...</a>
Related: <a href="https://news.ycombinator.com/item?id=42682602">https://news.ycombinator.com/item?id=42682602</a><p>Just finished the book recently. It's very insightful. As someone who considers himself good at debugging* and is still trying to improve debugging skills and efficiency, I view this book (and similar resources like this article) as a guide that also helps me reflect on what could have been done in a better way next time.<p>* Multiple times, I helped others find the root cause of a bug after they spend hours at it and have no clue what is happening
What I missed here was the importance of keeping careful notes as you go. What <i>exactly</i> happened when we constructed that weird input and commented out line 353? What hypotheses are we entertaining? Can we rule out any of them based on our evidence? It’s very easy to dupe yourself if you’re doing it all in your head.
What do you think will be the last skills/jobs to go obsolete?<p>I think it's "Wanting the right thing" (This includes figuring out what the right thing is) and "Being able to articulate your wish clearly" (This includes clarifying your thoughts).
Well “building” is also troubleshooting, fixing a problem. Just in a bit more general level: ideally it is not fixing a “small” well-defined problem in software, but bigger and fuzzier problem in the real world: the thinking process and tooling is quite the same. Of course many devs dont think of it like that, they just try to fulfill given requirements without understanding real problem they troubleshoot. Actually a lot of software “builds” are really troubleshooting attempts on top of other software also, which makes that border even fuzzier.
Half the comments here are nitpicking the car mechanic analogy (naturally), the other half are complaining about the site shitting the bed.<p>Yes, debugging is important, and too many people can't do it, which is unsettling considering how many bugs those people are putting into the code in the first place.
Brendan Gregg's USE method is for performance troubleshooting but could work in any situation (broken is just the worst performance, right?)<p><a href="https://www.brendangregg.com/usemethod.html" rel="nofollow">https://www.brendangregg.com/usemethod.html</a>
> I’ll define troubleshooting as systematically determining the cause of unwanted behaviour in a system, and fixing it.<p>Or debugging and understanding the reason why a system isn’t behaving as expected. And pinpointing the part of the code that causes the behaviour that is not desired.<p>In another field—In IT Service Management (ITSM)—there is the distinction between incidents and problems. If you see many incidents coming in that are related, you sit down and start doing a root-cause analysis, basically a form of debugging. Or troubleshooting.<p>So yes, this is a skill that is timeless.
"It’s sometimes more effective to approach troubleshooting slowly, thoughtfully, meditatively, even when in a hurry."<p>Shouldn't this say "... than when in a hurry."?
I feel like the way to think about troubleshooting is to think about it as an umbrella encompassing reliability and quality engineering in software. If you can find ways of showing how reliability and quality of a software can be broken and how it can be improved (simultaneously), then you have a career to make.<p>Don't wait for stuff to break and react. Be proactive and find ways to demonstrate how it can break and how to fix it.
> even seemingly simple systems are infinitely complex. (As Carl Sagan put it: “If you wish to make an apple pie from scratch, you must first invent the universe.”)<p>Wikiquote cites the book “Cosmos” page 218 as the source of this cool quote. <a href="https://lccn.loc.gov/80005286" rel="nofollow">https://lccn.loc.gov/80005286</a>
When I'm stumped troubleshooting production problems, I try to think how I can get more information out of the system.<p>Exporting telemetry events with a wide set of attributes to an observability platform is a great approach which can provide an extensible way to expose additional information about the events.
> Don’t assume it’s complicated<p>There are problem areas where it is a lot easier to assume everything is a 10/10 monster.<p>If you start every journey with "power cycle the device" and always wind up with a bridge call between 3 vendors, you might as well get the bridge warmed up the moment something throws a warning.<p>Oftentimes, getting someone on the phone can be a bit of a circus act regardless of what the contracts say. Over reacting early on can minimize total time to resolution.
When the CEO's vibe-coded slop gets chucked over the wall to become someone else's problem once completed in rough prototype form, and the ensuing bugs and scalability/reliability issues manifest, troubleshooting is going to be a more valuable skill than ever!<p>We'll get paid peanuts for it, but hey, we should be thankful for the work in the first place!