If you succeed you will fail

As an engineer, I&#x27;ve never personally understood the desire to release anyway, even when a system has known critical deficiencies like this one. Sure, perfect is the enemy of good, but I&#x27;d also argue that inoperable is the enemy of good - it only serves to erode trust in your product, if your team chose to release an unusable product.<p>Some years ago, I worked on a system not quite as bad as what the author described, but close. We released a new product with a known, quite bad security vulnerability (I&#x27;d made sure our product team was _extremely_ aware of this), as well as no monitoring to speak of. The deadline had been communicated for around one year, but nobody had ever really discussed the significance of the date, other than it was what we were all death-marching to, and we needed to deliver.<p>What did that date turn out to be? The head of product management&#x27;s birthday, which was revealed to the rest of the company on the highly-celebrated the launch date. People were just kissing ass. I left several months later.<p>It feels unconscionable to me that a company could have launched an incomplete, insecure, customer-facing product just to give a birthday gift to a leader, but I suspect this sort of thing is common.

&quot;Fall back on blaming the process when failure comes around and avoid ever pointing fingers or owning it.&quot;<p>Some organisations can do this, but I&#x27;ve see plenty that might outwardly try to avoid pointing fingers, but you can tell that despite warnings given, they do blame.<p>I told several leaders that one of their systems was literally on the brink and we were fighting fires on it every other day, and the processes in place were horribly broken.<p>12 months later shit hit the fan, my feedback was that I wasn&#x27;t proactive enough, and they basically threw me and my team under the bus.<p>This despite budget under cutting, limits in hiring, enormous optimisation, education of other teams, research. Just getting &#x27;No&#x27; or no real outcomes all the time on any escalation.<p>The leaders did this serially across the business, and lacked ownership on this aspect to even give people the resources and autonomy to do anything about it, yet would come looking when it came time, to throw other teams under the bus like this

&gt;I also learnt to only become as involved and care as much as the customer. If they aren’t willing to go that extra mile then why should I?<p>There was probably a good portion of that army of ops and developers that felt the same way. And even if you can move mountains to point fingers and clean out all of the bad people, train the average ones, and keep the good ones around, you will likely get little compensation out of it, so no one does that. Even then, it&#x27;s hard to blame incompetent people getting jobs and holding on to them. There&#x27;s a lack of standards for skill and training so naturally people slip through the variety of bespoke vetting processes that companies come up with. There&#x27;s also some economic reasons why people hold on to jobs for far longer than they should, even if they know they&#x27;re bad at it.<p>Large amounts of bad complexity slowly erodes morale, but bad decisions like this one can set it on fire. Accruing bad complexity may not have any solid failures to trigger reassessment, which is why it often becomes a problem and stays a problem. It&#x27;s an obvious problem for new people with fresh perspectives, while incumbents might have gotten used to it.

&gt; I learnt a few things from this. The first is, some companies or organisations within them need failure in order to progress.<p>Encountered a similar situation a few years back while working for a government client. My advice to the team was that we needed to “let the train wreck happen.”

Poor management from executives seems to be the norm.<p>- They failed to get the internal developer to buy in to the project or really work with the other team.<p>- They insisted on launching even though it was explained that the system wasn&#x27;t ready.<p>- Insisted on having a launch party when failure was expected.

Companies usually don&#x27;t have all their parts aligned or even functioning properly. And <i>you</i> cannot change that.<p>Your work isn&#x27;t just the product, it is navigating an imperfect structure to achieve what you need to, despite the fact that the imperfect structure is the one contracting the product&#x2F;service, and is (mostly unwittingly) standing in the way of that.<p>It&#x27;s not fair, it&#x27;s not fun. It is what it is.

Worth considering: why did the managers ignore the writer’s warning of the apocalypse when the system went live?<p>One possibility that comes to mind is that they’ve heard dire warnings before from other engineers which have turned out to be exaggerated.<p>Engineers do have Chicken Little tendencies sometimes, and the long term results can be similar.

A strangely, poorly performing authentication system with a good one a few weeks behind the launch. An oddly configured circular fail over strategy. The thing I’ve noticed about entrentched employees who don’t perform is that while they are not so good at creating&#x2F;building things, they are really good at pulling strings to make themselves look good at the expense of others.<p>I don’t think it was a great look for them that you were brought in to build this, and once you start pointing out flaws in other people’s systems you become a walking target. I’ve made this mistake too many times in the past, determined not to do it in the future.

&gt; I also learnt to only become as involved and care as much as the customer. If they aren’t willing to go that extra mile then why should I?<p>This is what I use at work to keep myself sane. If the highly paid VPs, directors, managers don&#x27;t care about something, why should I?<p>This leads to a lot of friction with manager where they want to put blame on individuals, months after they ignored the warnings. But realistically, it is their fault and they will get fired soon enough if they try to make everything a problem of engineers.

I&#x27;ve seen versions of this, anytime someone builds on lambda and then points it at something not built on lambda.<p>The impedance mismatch is usually too great and speaks to some serious deficiency at the architecture stage.

reminds me of the similar sentiment that a failure may be required in order to get all parties to accept that a change&#x2F;improvement is needed.

I have worked with well over a dozen organizations with ERP systems from SAP, Oracle and IBM. When organizations are large enough to <i>need</i> such systems, the management has grown to the level of being driven by politics and hubris. Any warnings by technical staff are rejected because they contradict the management&#x27;s belief that they know better.

I tell people that we do great work for its own sake; without parental affection; knowing that, brilliant or bogus, the underside of the bus beckons.

Cause one person with the support of c level makes a decision, we can all stand back and watch 70+ years of success crash and burn in flames.

Just commenting on the &quot;If you succeed you will fail.&quot;.<p>Failure is envitable and to succeed at something you have had to have had experience encountering a problem. If you succeeded first time encountering a problem it is likely you&#x27;ll never know the solution to the problem and thus it is envitable failure will eventually become of it.<p>It is enough to say success is built on a mountain of failures. Only difference between something being a failure and it becoming a success is that attempts made to produce a desired result.

You keep going until something stops you. So if you do everything right and solve every problem thrown at you, that just means you&#x27;re going to push harder. Eventually something&#x27;s going to break. This isn&#x27;t &#x27;failure&#x27;, it&#x27;s just how development, business, really everything works.

It’s interesting how such things impact an employe.<p>There are times where things have to fail in order for change to happen, but employees also generally want to make things that are meaningful, and work. It makes the process to inevitable failure really frustrating &#x2F; demoralizing for the employees.

The sooner that you find success the sooner that you&#x27;ll fail So don&#x27;t, don&#x27;t, don&#x27;t Don&#x27;t play the chords of fame

I love watching fake deadlines go past.<p>Good, cheap, fast, pick at most two. Maybe one. Sometimes none. Occasionally negative.

Where was architecture in scoping?

Not to be <i>that</i> guy but that should be a comma and not a semicolon.