Right. Commenting on a very specific part of this with an anecdote. I did some integration with a ‘neo bank’ a few years ago where the CTO said testing and staging envs are a lie. I vehemently disagree(d) but they were paying to only test on production. I guess you can guess what happened (and not because of us as we spent 10k of our own money to build a simulator of their env; I have some kind of pride); testing was extremely expensive (it is a bank so production testing and having bugs is actually losing money... also they could not really test millions of transactions because of that so there were bugs, many bugs, in their system...), violated rules and the company died and got bought for scraps.<p>I understand the sentiment but I agree with 2, 3 and 6 of this article, the rest is, imho, actually dangerous in many non startup cases.<p>Example; Simple is always better IF you can apply it; a lot of companies and people you work with do not do so simple. A lot of companies still have SOAP, CORBA or in house protocols and you will have to work with it. So you can shout from the rafters that simple wins; you will not get the project. That can be a decision but I do not see many people who finally got into a bank/insurer/manufacturer/... go ‘well, your tech is not simple in my definition so I will look elsewhere’.<p>It is a nice utopia and maybe it will happen when all legacy gets phased out in 50-100 years.
This article makes a lot of assumptions that only hold true in a very specific set of circumstances:<p>- that it’s possible for the team developing the product to deploy or monitor it (example cases where it isn’t: most things that aren’t web based such as desktop, most things embedded into hardware that might not yet exist etc.)<p>- that <i>if</i> you can deliver continuously, customers actually <i>accept</i> that you do. Customers may want Big Bang releases every two years and reject the idea of the software changing the slightest in between.<p>- not validating a deployment for a long time before it meets customers is also only ok if the impact of mistakes is that you deploy a fix. If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested.<p>My point is: if you are a team developing and operating a product that is as web site/app/service and you are free to choose if and when to deploy, then most of the article is indeed good advice. But then you are also facing the simple edge case among software deployment scenarios.
Cool article, enjoy the summary of relevant knowledge that's been passed around various circles.<p>I do disagree with:<p>> Environments like staging or pre-prod are a fucking lie.<p>You need an environment that runs on production settings but isn't production. Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME.<p>There's just so many classes of bugs that are easily caught by some sort of pre-prod environment, including stupid things like "I marked this dependency as development-only but actually it's needed in production as well".<p>Development environments are frequently so incredibly far removed from production environments that some sort of intermediary between production is almost always so helpful to me that the extra work involved in maintaining that staging environment is well worth it.<p>It's not the same as production obviously, but it's a LOT closer than development.
Some good points but some controversial ones.<p>I think a manual QA team is very valuable. Sure the tests pass but what if the UI is confusing or disorienting. QA can be user advocates in a way a unit test can't be. I work in games so maybe it's just a squishier design philosophy but you can't unit test fun.<p>I also don't understand the worry about other environments. If you're automating deployments how is another environment added work? Shouldn't it be just as easy to deploy to?
I think I disagree with this about 100%. Sure, production is what it is all about in the end. But how do you know the letters you just typed are going to be any good in production? They might just crash and burn there. That is why we need all those quality gates. The sooner and the farther removed from production that you discover a problem the easier it is to fix.<p>Regarding the 'buy vs build' I think buying software is one of the most risky things that you can do. Since it cost money you cannot then say 'o well, i guess it just did not pan out, let us just not use it'. Now you are kind of married to the software. And some of the worst software out there is paid for. E.g., jira vs. redmine. This is actually a bit ironic considering the fact that I actually am writing software in my job that is sold.... O well, it actually is sold as a part of a piece of hardware, so it is not really sofware as such.....<p>Regarding the last point, failure can be made uncommon if a relatively safe route to production is available, starting with a language that verifies the correct use of types, automated tests that verify the correctness of code, a testing environment that one attempts to keep close to what production is like and so on. Getting a call that production is not working is the event that I am trying to prevent by all means possible, and I think research would be able to show that people who get fewer calls, not just because production is failing, but in general, fewer calls regarding whatever subject, will live longer and happier.
He lost me at “non production environments are bullshit”.<p>In dev you can break almost anything, no biggie. In stage if you break something, great just don’t deploy it to prod. If you break something in prod, well ... you may end up going below SLA and may legit lose money and your customers trust.<p>Don’t YOLO into prod. Build reliable shit.
Non-production environments are useful for more than testing application code. Changing underlying infrastructure (Upgrading a database, networking shenanigans, messing around with ELB or Nginx settings) requires testing too. Having the same traffic / data shape in pre-prod is not as important.
Radical! Some counter-points, though.<p>- Infrastructure as code and schemas as code make it easier to keep environmental parity, because everything can be rolled back/forwards/reset with easy source control and CD operations. Visual environment diffing and drift detection can make this even easier.<p>- Make your stage and prod into a blue-green situation, where if stage is ready to go, you flip users onto it. I can guarantee your stage and prod will both be respected as prod then. Failing that, just add load/stress tests to stage to make it more prod-like.<p>- Non-prod environments and attention are not necessarily debt, but they are expensive insurance premiums. You should only pay those premiums if you need the insurance. It's about risk management.<p>- As time passes, the people who wrote a specific part of a system don't know it anymore, so having them babysit 'their' code in production has diminishing returns. On the other hand, having a systems quality team who have a broad mandate to bugfix, put in preventative measure, reduce technical debt, improve observibility and establish good patterns for developers to do these things, can enabled these things to actually happen, when just telling devs who are busy making features that they should happen often doesn't make them happen. Also there are devs who enjoy creating new things, and others who love trouble-shooting and metrics.
What exactly is would be the disadvantage of running something in stating environment before running it "for real" in production? I'm assuming the staging environment is an exact clone of production (except reduced size: fewer app servers + smaller DB instance)?<p>I understand the deploy-often-and-rollback-if-there-is-a-problem strategy, but certain things like DB migrations and config changes are difficult to rollback, so doing a dry run in a staging environment seems like a good thing...
It's funny in my career I've observed similar development styles. But I always just thought of this as great/good developers verses average/mediocre developers. the A+ coders would always make their code very easy to access, deploy from a user standpoint, debug, read etc. The mediocre guys would wait for someone else to hit a landmine beefore fixing something that was obviously wrong.
While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment.<p>For some of our customers, we operate 2 environments which are both effectively production. The only real difference between these is the users who have access. Normal production allows all expected users. "Pre" production allows only 2-3 specific users who understand the intent of this environment and the potential damage they might cause. In these ideal cases, we go: local development -> internal QA -> pre production -> production actual. These customers do not actually have a dedicated testing or staging environment. Everyone loves this process who has seen it in action. The level of confidence in an update going from pre production to production is pretty much absolute at this point.<p>The amount of frustration this has eliminated is staggering. At least in cases where we were allowed by our customers the ability to practice it. For many there is still that ancient fear that if we haven't tested for a few hours in staging that the world will end. For others, weeks of bullshit ceremony can be summarily dismissed in favor of actually meeting the business needs directly and with courage. Hiding in staging is ultimately cowardice. You don't want to deal with the bugs you know will be found in production, so you keep it there as long as possible. And then, when it does finally go to production, it's inevitably a complete shitshow because you've been making months worth of changes built upon layers of assumptions that have never been validated against reality.<p>This all said, there are definitely specific ecosystems in which the traditional model of test/staging/prod works out well, but I find these to be rare in practice. Most of the time, production is hooked up to real-world consequences that can never be fully replicated in a staging or test environment. We've built some incredibly elaborate simulators and still cannot 100% prove that code passing on these will succeed in production against the real deal.
I agree with most of the points, but I have serious caveats on the first two.<p>1. No, the engineers should not by default be on call; the owners of the product are the first call line. If they're not engineers or if they're engineers but don't have enough time to deal with all incidents–in short, if they need to delegate–they better be willing to pay <i>very</i> generously for the extra hours of on call duty.<p>2. No, hosted is not better than open source, both for philosophical and operational reasons: mostly, you become subject to the whims of the provider. A good compromise is <i>hosted open source</i> solutions, which at least takes you half way to a migration, if the need for one comes up.<p>That aside, I very much agree on everything else.
I agree with most of this but the point about QA gating deploys should be amended. A 5 minute integration test on a pre-flight box in the production environment by the deploying engineer is a form of QA, and can catch a lot of issues. It shouldn’t be considered anti pattern. Manually verifying critical paths in production before putting them live is about the best thing you can do to ensure no push results in catastrophic breakage.<p>Without such a preflight box, or automated incremental rollouts, you are kind of doing a Hail Mary, since you are exposing all users immediately to a system that has not been verified in production before going live.
I agree with most everything said in the article but with a big condition. If I as an engineer am responsible for everything the author says I should be responsible for then I want total control of the tech stack and runtime environment.
> 2. Buy Almost Always Beats Build<p>Strongly disagree with that, well maybe it is a good idea when you are over founded by VC where cost of money is equal to zero and you don't want to master what you are working on but in all other cases this is wrong, you shouldn't rebuild everything from scratch but creating a company is not the same as playing with LEGO<p>And this is the same argument as saying you should have everything in AWS because if you self host you will have to hire devops engineer