I've gone back and forth on this. Yes - sometimes your worry is completely unfounded, but this article lost me here:<p>>Untangling messy parts of the codebase, identifying and removing dead code, cleaning up build, deploy and monitoring systems - these tasks often feel overwhelming because they can overwhelm you, sucking up days for no real gain.<p>Those aren't the risks I'm thinking about when I'm worried about problems like this. Difficulty/time is not the concern. Unknown-unknowns are the concern.<p>Upgrading our NodeJS version takes about 10 seconds with Elastic Beanstalk on AWS, but we're not on the latest version, even though it seems to run without issue and our tests pass. Why? Because maybe it'll expose some race case that would only appear in production. And worse even, maybe that race case will affect a lot of people <i>sometimes</i> in an extremely hard to reproduce way. Maybe it'll break something that isn't apparent for weeks.<p>We had a bug where <i>some</i> SMS notifications weren't being sent in production, and it was hurting our retention metrics for a couple months before one of our engineers stumbled across it. It was caused by a maintenance upgrade of one of our very few dependencies. In a small company or on a new product, you don't have the kind of bulletproof reporting to <i>know</i> when something like that is broken - your metrics are moving around quite a bit by default.<p>Code that works and has been working for ages without issue is code that I am not interested in changing unless I absolutely have to. Leaving/avoiding code that hasn't been changed in forever is sometimes the best option to ensure stability.<p>This all being said, I agree that this worry can be carried too far. Like I said - I've gone back and forth on this. Fundamentally it's a tricky line to walk.
I agree with the points made. I would add one caveat: The culture of your work environment has a huge bearing on these decisions.<p>Reminds me of something I read, like an illegal life tip; Don't fix an unknown problem because if it goes wrong people often won't understand or care about your good intentions and you will end up with blame (blame culture).<p>Whereas if a problem where to occur on production, and you were to fix it, well then you would be "hero".<p>Not trying to advocate for any particular way but just trying to reframe the question in more of a political/human way as opposed to an engineering one.
I find that I'm often advocating for incremental improvements when doing code-reviews. Knowing that a wider view will be taken anyway by me will change the attitude of my peers when approaching problems.<p>Another benefit of code-reviews is that they share the risk. So making a bigger change will feel safer to the individual.
> What if they succeed, or worse still succeed easily? That would reveal that you weren’t good enough to solve the problem<p>I would simply congratulate the engineer for seeing something that I didn't.<p>Every once in a while something like this happens to me, and I found that being humble is a good way to relax at work.
Yup, this is the psychology I’ve seen at work, and it’s weird because my own psychology is the opposite. When I’m coding by myself or on a team with code ownership, I naturally seek 100% understanding of the code I’m working on and worry about anything less. I am drawn to the thorny parts of a codebase (that nobody understands because ten different people have been playing bug whack-a-mole) so I can clean them up. Just as if one corner of a room had mold or bugs in it I would want to clean that up upon moving into a group house.
Anxiety Reduction: I embed realtime monitoring in my code (high performance, specialized HTTP servers) using Statsd and Graphite (Graphana). I have to monitor 132 servers around the world all running the same software. Realtime stats and graphs completely removes a whole level of anxiety about "what is going on with my code". Actively measuring things like.<p>1) Transactions per second
2) Total time to process HTTP request and return response
3) Timing of specific sections of code
4) Exception counts
5) Specific function counts (get as detailed as you need here)
6) Response time as viewed from an external requester (with percentile breakdown)
7) Version numbers
8) Anything you feel is important to know about your software<p>I always deploy canaries and compare the stats of the canary box to the ones running version canary-1. Any difference in stats must be investigated and explained as valid before a new version can be deployed beyond canary.<p>The greatest tool I have ever created for regression testing, I call "A/B test". I record incoming network traffic, right off the wire using "ngrep". I have a tool that plays back this traffic to multiple different destinations and compares the response from each. Any difference in response between old version and new version must be explained and not caused by bugs. This tool also reads log files and other forms of output and does the same type of comparison. Obviously some things are always different like random numbers in responses, unique cookies, timestamps can differ by 1 second sometimes etc. Those things are removed before comparing.<p>This tool almost 100% guarantees no unintended change of behavior slips through a new release. The only way something could slip through here is if some odd request that wasn't tested causes it, or something in the data that we don't compare (due to it always being different like random number) causes something. I run 10 million requests through this A/B test and a "no unexplained difference" result is required before fully deploying.
> Engineers avoid work that needs doing because they’re afraid of becoming entangled with a nightmarish task<p>not most engineers I know; they need to make money and will just do whatever to make that money.
I've been doing this for a while now and I just can't get anything done if such anxiety takes over, although I'm a junior developer I'm often found in situations to make decisions like a senior dev.
And I don't know what to do about it, which makes it worse.