I don't know if this is O-Ring theory as much as a "six sigma" style process. The quality of each output depends on the quality of the inputs, and imperfections grow exponentially when things fall out of tolerances.<p>The whole point of DevOps is really to get your software developers to share in the pain that certain design / development choices can cause. In the absence of some DevOps-like process, dev teams will naturally shovel all their tech debt on the operations team. Rather than build their software to handle multi-tenancy, they'll just say "Eh, deploy another VM with a different configuration". Not that that's necessarily the wrong answer, but because it's less work for them, they don't take into account how hard it will be for the operations team to manage. If the dev team is responsible for both building the software, the tools used to manage the deployment, and responsibility for incident resolution, they'll make decisions that are better for the company and its customers.<p>That's my take, anyway.
This is what people mean when they talk about "technical debt". Many of us have realized that when mistakes are not fixed early, we end up doing 10x, 20x, and even 200x the amount of work in dealing with the consequences of those mistakes.<p>In Software Development good tools like an IDE with static code analysis and good practices like unit testing + integration testing + functional testing + automated acceptance tests help us to avoid that 200x downside.<p>DevOps is not all software development, but part of it is, and so it should respond in the same way to good tools, and good test practices. For the rest of it, there are lots of old practices that do make a difference such as having regular backups of everything, having live-live standby servers, hiring a DBA to clean up the mess in your data models, NEVER patching the database except using code that has been unit tested and which generates a script to undo everything that it has patched along with a hash check of the database to guarantee that the undo step worked. And so on.<p>Maybe the cute moniker "O-ring economics" will help to explain it to management types better but the jury is still out on that, IMHO.<p>But what manager would NOT want everybody to work as a team, to learn from their mistakes, and to constantly up their game, as a team? Call it Agile or call it something else, but this is what is proven over the last 70 years or so (perhaps even longer).
Excellence in development and exellence in operations does not necessarily overlap.
IMHO the whole "devops" term is bogus since it's just the worst of both worlds.<p>You want experts at both ends of the spectrum, not a developer who can do some sysadmin tasks but not too well or a operational engineer who can write some code but not too well.
Acknowledge that there are different fields and you need experts in both. Forget about aiming for a hybrid - it doesn't work. You'll just get incompetents who are able to pose as being good at both sides.
Uh, that isn't the "O-Ring Theory" I would have expected.<p>The Challenger failed because, while the O-Rings were within tolerances, they shouldn't have been varying <i>at all</i>. So because they were varying unexpectedly, they inevitably failed.<p>I would claim the corresponding devops theory is "measure everything, know what is measurement noise and what is user activity, and eliminate everything else before it causes an outage".<p>(from Wikipedia: "In one example, early tests resulted in some of the booster rocket's O-rings burning a third of the way through. These O-rings provided the gas-tight seal needed between the vertically stacked cylindrical sections that made up the solid fuel booster. NASA managers recorded this result as demonstrating that the O-rings had a "safety factor" of 3. Feynman incredulously explains the magnitude of this error: a "safety factor" refers to the practice of building an object to be capable of withstanding more force than the force to which it will conceivably be subjected. To paraphrase Feynman's example, if engineers built a bridge that could bear 3,000 pounds without any damage, even though it was never expected to bear more than 1,000 pounds in practice, the safety factor would be 3. If a 1,000 pound truck drove across the bridge and it cracked at all, even just a third of the way through a beam, the safety factor is now zero: the bridge is defective.")
This is very true in the world of business intelligence and analytics. A single breakdown in the data pipeline from source to report/dashboard affects the entire system's value. "Garbage in, garbage out" is a maxim. If the ETL pipeline does not clean up the data and correct errors, it can make dashboards nearly worthless.
Let's assume the leadtimes in the pipeline differ dramatically. l_1 ... l_n-1 take just minutes and l_n takes some days.<p>Maybe its a huge simulation or learning of a neural network.<p>In this theory the leadtimes are getting normalized and multiplied.<p>But in this pipeline, the process n is overweighting all the other processes, so that the leadtimes of the other processes are almost irrelevant compared to the leadtime of process n.<p>a. Lets assume leadtime of process 1 is missed by 50%. So it takes an additional minute.<p>b. Compared to process n, missing leadtime by 50%, taking an additional week.<p>Expected Quality would be the same for a and b.
But the overall leadtime will differ in one week.