This is pretty common actually<p>At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod<p>That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better<p>This is how we do it at my current team too…for all the same reasons that OP states
This article has some very weird trade-offs.<p>They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso.<p>They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. If they didn't hit those roadblocks in managing their staging environments, presumably they would be using them.
I don't see how this can scale beyond a single service.<p>Complex systems are made of several services and infrastructure all interconnected.
Things that are impossible to run on local. And even if you can run on local, the setup
is most likely very different from production. The fact that things work on local
give a little to zero guarantees that they will work in prod.<p>If you have a fully automated infrastructure setup (e.g: terraform and friends), then
it is not that hard to maintain a staging environment that is identical to production.<p>Create a new feature branch from main, run unit tests, integrations tests. Changes are
automatically merged in the main branch.<p>From there a release is cut and deployed to staging. Run tests in staging, if all good,
promote the release to production.
This is good insofar as it forces you to make local development possible. In my experience: it's a big red flag if your systems are so complex or interdependent that it's impossible to run or test any of them locally.<p>That leads to people <i>only</i> testing in staging envs, causing staging to constantly break and discouraging automated tests that prevent regression bugs. It also leads to increasing complexity and interconnectedness over time, since people are never encouraged to get code running in isolation.
I don't think you can take infrastructure seriously without a staging environment. For many companies that is fine - they don't have significant infrastructure to maintain (or just don't maintain the infrastructure they have).<p>I work on a team that maintains our database layer and the lack of a staging environment is incredibly painful. Every test has to be done in production and massive effort needs to be taken to proceed safely. With a staging environment you can be more aggressive and come up with a solid benchmark and test suite to gain confidence rather than having to data collect in prod
The short answer appears to be "we are cheap and nobody cares yet."<p>It's easy to damn the torpedoes and deploy straight into production if there's nobody to care about, or your paying customers (to the extent you have any) don't care either.<p>Once you start gaining paying customers who really care about your service being reliable, your tune changes pretty quickly. If your customers rely on data fidelity, they're going to get pretty steamed when your deployment irreversibly alters or irrevocably loses it.<p>Also, "staging never looks like production" looks like a cost that tradeoff that the author made, not a Fundamental Law of DevOps. If you want it to look like production, you can do the work and develop the discipline to make it so. The cloud makes this easier than ever, if you're willing to pay for it.
This is a pretty weird article. Their "how we do it" section lists:<p>- "We only merge code that is ready to go live"<p>- "We have a flat branching strategy"<p>- "High risk features are always feature flagged"<p>- "Hands-on deployments" (which, from their description, seems to be just a weird way of saying "we have good monitoring and observability tooling")<p>...absolutely none of which conflict with or replace having a staging environment. Three of my last four gigs have had all four of those <i>and</i> found value in a staging environment. In fact, the often <i>help</i> make staging useful: having feature-flagged features and ready-to-merge code means that multiple people can validate their features on staging without stepping on eachother's toes.
<i>I'm assuming this is not an April Fools' joke, and my comments are targeted at the discussion it sparked here anyway.</i><p>A flat branching model simplify things, and the strategy they describe surely enables them to ship features to production faster. But the risks I see there:<p>- who decides when a feature is ready to go to production? The programmer who developed them? The automated tests?<p>- features toggleable by a flag must, at least ideally, be double-tested -- both when turned on and off. Being in a hurry to deploy to production wouldn't help on that;<p>- OK, staging environments aren't in parity with production. But wouldn't they be better than the CD/CI pipeline, or developer's laptop, testing new features in isolation?<p>- Talking about features in isolation: what about bugs caused by spurious interaction between two or more features? No amount of test would find them if they only test features in isolation
If you can, provide on-demand environments for PRs. It's mostly helpful to test frontend changes, but also database migrations and just demoing changes to colleagues.<p>If you have that, you will see people's behaviour change. We have a CTO that creates "demo" PRs with features they want to show to customers. All all the contension around staging as identified in the article is mostly gone.
Not endorsing this point blank but.. One positive side effect of this is that it becomes much easier to rally folks into improving the fidelity of the dev environment, which has compound positive impact on productivity (and mental health of your engineers).<p>In my experience at Big Tech Corp, dev environments were reduced to low unit test fidelity over years, then as a result you need to <i>iterate</i> (ie develop) in a staging environment that is orders of magnitude slower (and more expensive if you're paying for it). It isn't unusual that waiting for integration tests is the majority of your day.<p>Now, you might say that it's too complex so there's no other way, and yes sometimes that's the case, but there's nuance! Engineers have no incentive to fix dev if staging/integration works at all (even if super slow) so it's impossible to tell. If you think slow is a mild annoyance, I will tell you that I had senior engineers on my team that committed around 2-3 (often small) PRs per month.
> We only merge code that is ready to go live.<p>In their perception, is the rest of tech industry gambling in every pull request that some untested code would work in production?<p>I work at a large company. We extensively test code on local machines. Then dev test environments. Then small roll out to just a few data centers in prod bed. Run small scale online flight experiments. Then roll out to the rest of prod bed.<p>And I've seen code fail in each of the stages, no matter how extensively we tested and robustly code ran in prior stages.
Good monitoring, logs, metrics, feature flagging (allowing for opening a branch of code for a % of users), blue/green deployment (allowing a release to handle a % of the user's traffic) and good tooling for quick builds/releases/rollback, in my experience, are far better tools than intermediate staging environments.<p>I've had great success in the past with a custom feature flags system + Google's App Engine % based traffic shifting, where you can send just a small % of traffic to a new service, and rollback to your previous version quickly without even needing to redeploy.<p>Now, not having those tools as a minimum, and not having either staging environment is just reckless. No unit/integration/whatever tests are going to make me feel safe about a deploy.
We used to believe staging environments are not important enough. If you believe that then I would argue that you have not crossed a threshold as an org where your product is critical enough for you consumers. The staging environment or any for that matter just acts as a gating mechanism to not ship crappy stuff to customers.
You cannot have too many gates, then you would be shipping lates but with less number of gates you end up shipping low quality product.<p>Staging environment saves unnecessary midnight alerts and easy to catch issues that might have a huge impact when a customer has to face it. I wouldn't be surprised if in few quarters or a year or so they would have an article about why they decided to introduce a staging environment.
This sounds like something I would write if a hypothetical gun was pointed at my head in a company where the most prominent customer complaint was that time spent in QA and testing was too expensive.<p>I have zero trust in any company that deploys directly from a developer's laptop to production, not in the least starting with how much do you trust that developer. There has to be some process right?
> Pre-live environments are never at parity with production<p>Then you fix that particular problem. Infrastructure as code is one idea just off the top of my head.
I don't have experience with the true CI he describes, but I do have experience with pre-production environments.<p>> "People mistakenly let process replace accountability"<p>I find this to be mostly true. When the code goes somewhere else <i>before</i> it goes to prod, much of the burden of responsibility goes along with it. <i>Other</i> people find the bugs and spoon feed them back to the developers. I'm sure as a developer this is nice, but as a process I hate it.
What I infer from the article is this company does not handle sensitive private data, or they do but are unaware of it, or they are aware of it and just handle it sloppily. I infer that because one of the biggest advantages of a pre-prod environment is you can let your devs play around in a quasi-production environment that gets real traffic, but no traffic from outside customers. This is helpful because when you take privacy seriously there is no way for devs to just look at the production database, or to gain interactive shells in prod, or to attach debuggers to production services without invoke glass-breaking emergency procedures. In the pre-prod environment they can do whatever they want.<p>Most of the rest of the article is not about the disadvantages of pre-prod, but the drawbacks of the "git flow" branching model compared to "trunk based development". The latter is clearly superior and I agree with those parts of the article.
> People mistakenly let process replace accountability<p>> We only merge code that is ready to go live.<p>This is one of the most off-putting things I have read on HN lately. Having worked on several large SaaS where leadership claimed similar stuff, I simply refuse to believe it.
We duplicate the production environment and sanitize all the data to be anonymous. We run our automated tests on this production-like data to smoke test. Our tests are driven by pytest and Playwright. God bless, I have to say how much I love Playwright. It just makes sense.
> We only merge code that is ready to go live<p>Cool story, but you don't _know_ if its ready until after.<p>Look, staging environments are not great, for the reasons described. But just killing staging and having done with it isn't the answer either. You need to _know_ when your service is fucked or not performing correctly.<p>The only way that this kind of deployment is practical _at scale_ is to have comprehensive end-to-end testing constantly running on prod. This was the only real way we could be sure that our service was fully working within acceptable parameters. We ran captured real life queries constantly in a random order, at a random time (caching can give you a false sense of security, go on, ask me how I know)<p>At no point is monitoring strategy discussed.<p>Unless you know how your service is supposed to behave, and you can describe that state using metrics, your system isn't monitored. Logging is too shit, slow and expensive to get meaningful near realtime results. Some companies expend billions taming logs into metrics. don't do that, make metrics first.<p>> You’ll reduce cost and complexity in your infrastructure<p>I mean possibly, but you'll need to spend a lot more on making sure that your backups work. I have had a rule for a while that all instances must be younger than a month in prod. This means that you should be able to re-build _from scratch_ all instances <i>and datastores</i>. Instances are trivial to rebuild, databases should also be, but often arn't. If you're going to fuck around an find out in prod, then you need good well practised recovery procedures<p>> If we ever have an issue in production, we always roll forward.<p>I mean that cute and all, but not being able to back out means that you're fucked, you might not think you're fucked, but that's because you've not been fucked yet.<p>its like the old addage, there are two states of system admin: Those who are about to have data loss, and those who have had data loss.
This makes sense. With a high-enough release velocity to trunk, a super safe release pipeline with lots of automated checks, a well-tested rolling update/rollback process in production, and aggressive observability, it is totally possible to remove staging in many environments. This is one of the popular talking points touted by advocates of trunk-based development.<p>(Note that you can do a lot of exploratory testing in disposable environments that get spun up during CI. Since the code in prod is the same as the code in main, there's no reason to keep them around. That's probably how they get around what's traditionally called UAT.)<p>The problem for larger companies that tend to have lots of staging environments is that the risk of testing in production vastly exceeds the benefits gained from this approach. Between the learning curve required to make this happen, the investment required to get people off of dev, the significantly larger amounts of money at stake, and, in many cases, stockholder responsibilities, it is an uphill battle to get companies to this point.<p>Also, many (MANY) development teams at BigCo's don't even "own" their code once it leaves staging.<p>I've found it easier to employ a more grassroots approach towards moving people towards laptop-to-production. Every dev wants to work like Squeaky does (many hate dev/staging environments for the reasons they've outlined); they just don't feel empowered to do so. Work with a single team that ships something important but won't blow up the company if they push a bad build into prod. Let them be advocates internally to promote (hopefully) pseudo-viral spread.
This is good practice, except that blue/green is not exactly what you want. You want a smart load balancer that can shuffle an <i>exact</i> amount of traffic to a new service with your new deploy version. It must then evaluate the new service for errors and metrics, and then do an increase in shuffling traffic, etc, until you reach 100% shuffled traffic, at which time the old services can be decommissioned.<p>If at any time the monitoring of logs or metrics becomes unusual, it must shuffle all traffic away from the new service, alert devs, and halt all deploys (because someone needs to identify the bad code and unmerge it, thus requiring rework for all the subsequent work about to be merged). This is called "pulling the andon cord".<p>It is sad that there's all these comments saying this doesn't work. This has been the best practice established by Etsy, Martin Fowler, and others in the DevOps community for... 10 years? I guess until you see it for yourself it seems unbelievable. It requires a radical shift in design, development and operation, but it works great.
We have review environments so there is an easy way to have a fairly persistent config to QA features, but our environment that's named staging is more of a historical artifact. It's basically the same as a review environment because we recognize that after testing that the feature works as intended, it's going out and, we may be surprised by real production use. Our test suite, which is kicked off after you hit the merge button takes about 10-15 minutes and build/deploy to Amazon ECS is 8 to 10 minutes so there is pretty quick feedback. We also use feature flags when possible, but most deploys are very granular and we generally don't worry if something passes our test suite which is currently about 6k tests. Once we decided that merge to main get deployed automatically our staging environment became just another environment, our velocity increased, security patches are deployed almost immediately and we mostly don't worry about launches.
Disclaimer: I worked for a major feature flagging company, but these opinions are my own.<p>This article makes a lot of valid points regarding staging environments, but their reasoning to not use them is dubious. None of their reasons are good enough to take staging environments out of the equation.<p>I'd be willing to be that the likelihood of anyone merging code that isn't ready to go live is close to zero. You still need to validate the code. Their branching strategy is (in my opinion) the ideal branching strategy, but again, that isn't good enough to take staging away.<p>Using feature flags is probably the only reason they give that comes to close to being okay with getting rid of staging, but even then, you can't always be sure that the code you've built works as expected. So you still need a staging environment to validate some things.<p>Having hands-on deployments should always be happening anyway. It's not a reason to not have a staging environment.<p>If you truly want to get rid of your a staging environment the minimum that you need to feature flagging of _everything_, and I do mean everything. That is honestly near impossible. You also need live preview environments for each PR/branch. This somewhat eliminates the need for a staging because reviewers can test the changes on a live environment. These two things still aren't good enough reason to get rid of your staging environment. There is still many things that can go wrong.<p>The reason we have layered deployment systems (CI, staging etc) is to increase confidence that your deployment will be good. You can never be 100% sure. But I'll bet you, removing a staging environment lowers that confidence further.<p>Having said all of this, if it works for you, then great. But the reasons I've read on this post, don't feel good enough to me to get rid of any staging environments.
If you are saying you don't have a staging environment, what you are really saying is that your company doesn't have any QA process.<p>If your QA process is just developers testing their own shit on their local machine then you are not going to get as much value out of staging.
> We only merge code that is ready to go live
> If we’re not confident that changes are ready to be in production, then we don’t merge them. This usually means we've written sufficient tests and have validated our changes in development.<p>Yeah I don't trust even myself with this one. Your database migration can fuck up your data big time in ways you didn't even predict. Just use staging with a copy of prod. <a href="https://render.com/docs/pull-request-previews" rel="nofollow">https://render.com/docs/pull-request-previews</a><p>Sounds like OP could benefit from review apps, he's at the point where one staging environment for the entire tech org slows everybody down.
They mention database as a factor not to have a staging env due to different size, but they don’t mention how they test schema migrations and any feature which touches the data which usually produce multiple issues, or even data loss.
This makes some sense for a single application environment. In our system, however, there are dozens of interacting systems, and we need an integration environment to ensure that new code works with all the other systems.
This probably also depends on your core business. If your product does not deal with real money, crypto, or other financial instruments and it is not serious if something goes wrong with a small number of people in production, this may work for you. It is probably cheaper and simpler.
Lots of products are not like that. I built a bank and work on stock exchanges. Probably not a good idea to save money by not testing as people get quite annoyed when their money goes missing.
I think we are missing some contexts here. I have been trying to find more information about them. From what I found [1] (hopefully accurate) it looks like they are a new team - Beta in August 2021 and just incorporated in this February.
The founder/CTO is a full stack developer. I speculate they are a very small team (1-2 developers at the most) and a relatively straightforward architecture. In that context I suspect it is quite feasible to go from local to production without going through staging: They are likely to have a self sustained stack that can be packaged; they don't have a huge database or collection of edge cases; they have few customers, low expectation in terms of service level; they don't have stakeholders to review and approve features done (they are their own bosses). I emphasize with where they are, I have been in the same place at some point. It will be interesting to see whether this is sustainable without staging, or for how long, as they grow in team and offering.<p>[1] <a href="https://www.indiehackers.com/product/squeaky" rel="nofollow">https://www.indiehackers.com/product/squeaky</a>
An important piece of context missing from the article is the size of their team. LinkedIn shows 0 employees and their about page lists the two cofounders so I assume they have a team of 2. It's odd that the article talks about the problems with large codebases and multiple people working on a codebase when it doesn't look like they have those problems. With only 2 people, of course they can ship like that.
How do you do QA? I mean, staging in our case is accessible by a lot of non technical people that test things automated test cannot test (did I say test?).
It seems like an April 1st troll (based on publication date), but I am assuming its not.<p>I can only say that this is a fairly poor decision from someone who appears knowledgeable to know better.<p>They could do everything they are doing as-is in terms of process, and just add a rudimentary test on a Staging environment as it passes to Production.<p>Over a long enough timeline it will catch enough critical issues to justify itself.
Isn’t the concept of a single staging environment becoming a bit dated? Every recent project I’ve worked on uses preview branches or deploy previews, eg what Netlify offers <a href="https://docs.netlify.com/site-deploys/deploy-previews/" rel="nofollow">https://docs.netlify.com/site-deploys/deploy-previews/</a><p>Or am I missing something?
This reads like a Pre-Mortem.<p>When they lose all their most important customers’ data because the feature flags got too confusing… they can take this same article and say:
“BECAUSE WE xxxx that led to YYYY.<p>In future we will use a Staging or UAT environment to mitigate against YYYY and avoid xxxx”<p>Saving time on authoring a Post Mortem by pre-describing your folly seems like an odd way to spend precious dev time
I use a somewhat similar approach for Pirsch [0]. It's build so that I can run it locally, basically as a fully fledged staging environment. Databases run in Docker, everything else is started using modd [1]. This has proven to be a good setup for quick iterations and testing. I can quickly run all tests on my laptop (Go and TypeScript) and even import data from production to see if the statistics are correct for real data. Of course, there are some things that need to be mocked, like automated backups, but so far it turned out to work really well.<p>You can find more on our blog [2] if you would like to know more.<p>[0] <a href="https://pirsch.io" rel="nofollow">https://pirsch.io</a><p>[1] <a href="https://github.com/cortesi/modd" rel="nofollow">https://github.com/cortesi/modd</a><p>[2] <a href="https://pirsch.io/blog/techstack/" rel="nofollow">https://pirsch.io/blog/techstack/</a>
> Pre-live environments are never at parity with production<p>Same with your laptops... and this is only true if you make it that way. Using things like Docker containers eliminates some of the problem with this too.<p>> There’s always a queue<p>This has never been a problem for any of the teams I've been on (teams as large as ~80 people). Almost never do they "not want your code on there too". Eventually it's all got to run together anyway.<p>> Releases are too large<p>This has nothing to do with how many environments you have, and everything to do with your release practices. We try to do a release per week at a minimum, but have done multiple releases in a single day as well.<p>> Poor ownership of changes<p>Code ownership is a bad practice anyway. It allows people to throw their hands up and claim they're not responsible for a given part of the system. A down system is everyone's problem.<p>> People mistakenly let process replace accountability<p>Again - nothing to do with your environments here, just bad development practices.
This sounds like an organisational issue, not a technical, and I predict that this simply won't scale organisational-wise. It sounds like they have given no thought about their platform architecture, deploy pipelines, testing strategies, ... It's probably not yet causing issues because they're working in a small team, but rectifying this later will be an absolute pita.<p>That said, at scale, having a big staging/test/... can be impossible, but then things are split up organisationally, each team managing/service group/... managing their own environments, being responsible for the reliability/stability and availability towards other teams.<p>Also, with service meshes it has become feasible to actually test in production so you can let select users end up on specific (test) versions of a certain backend service.
This works for services that are growing slowly in features or have few other services integrating with it. I’m not sure how this scales when there are multiple services across multiple teams with dependencies on one another where services are being rapidly developed with new features. At work, we have staging/pre-prod environments across most teams that my team works with so new features can be tested in staging and other teams can test integrating with it. This is also possible to do with just a production environment but requires some engineering effort to add feature flags and special headers indicating a request is from a team looking to try a new API.
All of their “problems” with staging are fixable bathwater that doesn't require baby ejection.<p>I avoid staging for solo projects but it does feel a bit dirty.<p>For team work or complex solo projects (such as anything commercial) I would never!<p>On the cloud it is too easy to stage.<p>To the point where I have teared down and recreated staging environment to save a bit of money at times because it is so easy to bring back.<p>The article says to me their not using modern devops practices.<p>It is rare a tech practice “hot take” post is on the money, and this post follows the rule not the exception.<p>Have a staging environment!<p>Just the work / thinking / tech debt payoff to make one is worth it for other reasons: including to streamline your deployment processes both human and in code.
I don’t see how this works when you have multiple external services you don’t control in critical code paths that you can’t fully test in CI.<p>The cost of maintaining a staging environment is peanuts compared to 30 minutes of downtime or data corruption.
I have a lot of questions, but one above all the others. How do you preview changes to non-technical stakeholders in the company? Do you make sales people and CEOs and everyone else boot up a local development environment?
I’m working at megacorp at the moment as contractor. The local dev, cloud dev, cloud stage, cloud prod pipeline is truly glacial in velocity even with automation like Jenkins, kubernetes, etc. it takes weeks to move from dev complete to production. It’s a middle manager’s wet dream.<p>I used to wonder why isn’t megacorp being murdered by competitors delivering features faster, but actually, everyone is moving glacially for the same reason, so it doesn’t matter.<p>I’m kinda reminded by pg’s essay on which competitors to worry about. I might be a worried competitor if these guys are pulling off merging to master as production.
A previous client was paying roughly 50% of their AWS budget (more than a million per year) just to keep up development and staging.<p>They were roughly 3x machines for live, 2x for staging and 1x for development.<p>Trying to get rid of it didn't work politically, because we had a cyclical contract with AWS where we were committing to spend X amount in exchange for discounts. Also, a healthy amount of ego and managers of managers BS.<p>In terms of what that company was doing, I'm pretty sure I could have exceeded their environment for 2k per month on hetzner (using auction).
It might be enough for this company, but if you are a big corporate, it's definitely not something to do. You cannot expect millions of consumers to just be ok with the fact that the mobile app is done because it's too hard to keep in sync staging and prod.<p>I am maintaining the infra for a big mobile app and our staging environment allowed us in the last year to have only two production incidents and they were not due to code source (networking).<p>I really recommend any serious business to at least try it and see by themselves the advantages
When I started at Twitter, a guy was proposing building a staging environment. At that point it would have cost about $2M, so it was within the realm of conceivable. I was a pretty immediate “no”, for all sorts of reasons. Keeping the data in reasonable shape would have been a big project all by itself, getting reasonable load on it would have been another, and then of course it’s a large environment you need to be on call for but that doesn’t have the priority production has. It’s just an all around bad idea.
No disrespect but you can do this for an analytics dashboard or a content web site with canaries (Facebook). May not be the best for high liability sites like financial systems.
This appears to be just a naming convention issue. All the potential problems of staging environments can occur, for the same underlying reasons, in the approach advocated here, but they don't happen in staging merely because there isn't anything called that.<p>Personally, I think the approach advocated here is feasible, and even necessary if you are operating at global scale, but I am skeptical of tendentious stories about how it makes a number of problems just disappear.
It really depends<p>Without staging environment, your chance of finding critical bugs rely on offline testing. Not all bugs can be found in unit tests, you need load tests to detect certain bugs that doesn't break your program from correctness perspective, but on latency/memory leakage front. And such tests might take longer time to run.<p>Staging slows things down, but it is intended, it creates a buffer to observe behavior. Depending on the nature of your service, it can be quite critical.
In one of my very first jobs in the mid 90ies there was also an incoming team we took over from a major competitor, who made the processes I introduced much simpler by removing dev testing, staging and CVS.
They preferred to work as root on the live servers with 80.000 customers. Development was apparently so much easier with immediate feedback.<p>I liked that so much, that I resigned, and found a much better job 2 years later.
I guess you could say bad cultural fit.
> Pre-live environments are never at parity with production<p>As a B2B vendor, this is a conclusion we have been forced to reach across the board. We have since learned how to convince our customers to test in production.<p>Testing in prod is usually really easy <i>if</i> you are willing to have a conversation with the other non-technical humans in the business. Simple measures like a restricted prod test group are about 80% of the solution for us.
If they're not a parity then you are doing CI/CD wrong and aren't forcing deploys to staging before production. If you set the pipelines correctly then you *can't* get to production without being at parity with pre-production.<p>> they don’t want your changes to interfere with their validation.<p>Almost like those are issues you want to catch. That's the whole point of continuous integration!
The only way that you can create stable and safe systems is by introducing processes to ensure that your systems are stable and safe. It doesn't matter how much personal responsibility you claim to take, you are going to make mistakes, and processes are the mechanism for limiting the damage of those mistakes. This is core to the best practices behind any safety critical industry, and is embedded in functional safety. The logic of this article appears to be "We just concentrate very hard to make up for not having a decent staging environment". Which is fine if no one cares if your stuff breaks.<p>>When there is no buffer for changes before they go live, you need to be confident that your changes are fit for production.<p>This is just completely wrong headed. It's like saying you should learn to tight rope walk 100 metres from the ground because it's going to make you concentrate on not falling more. The solution for making mistakes isn't to increase the fallout of those mistakes. You can absolutely build a culture where you value putting the onus on the developer to make sure they have a sense of responsibility for keeping master clean and working, without abandoning the processes that help mitigate when you fail to do that.<p>The funny thing is, that when you see articles saying the opposite of this, almost <i>always</i> they will also say "over the course of X months, our new staging environment caught Y additional bugs that would have impacted production". I'd love to see the same here - some actual data on how much they "We're just going to concentrate harder" impacts production.
One approach I’m experimenting with is that all services communicate via a message channel (e.g. NATS or Pub/Sub).<p>By doing this, I can run a service locally but connect it to the production pubsub server and then see how it effects the system if I publish events to it locally.<p>I could also subscribe to events and see real production events hitting my local machine.
At my previous job we had a single staging environment, which was used by dozens of teams to test independent releases as well as to test our public mobile app before release. That said, it never matched production, so releases were always a crapshoot as things suddenly happened no one ever tested. Yes, it was dumb.
This is how we work at fastcomments... soon we will have a shard in each major continent and will just deploy changes to a shard, run e2e tests, and then roll out to the rest of the Shards.<p>But if you have a high risk system or a business that values absolute quality over iteration speed, then yeah you want dev/staging envs...
I dunno if I'm getting older or if this is as silly as it seems.<p>You don't like pre-live because it doesn't have parity with production, so you use a developers laptop? What???<p>I stopped reading at that point because that's pretty indicative of either a specific niche or a poorly thought out problem/solution set
I currently have this with a client. When I was the only backend developer, the staging server setup worked perfectly because it was basically the master branch + 1 or 2 queued changes. Now that other people have joined the mix it's become more of a nuisance.
different business or organisational contexts have different deployment patterns and different negative impacts of failure.<p>in some contexts, failures can be isolated to small numbers of users, the negative impacts of failures are low, and rollback is quick and easy. in this kind of environment, provided you have good observability & deployment, it might be more reasonable to eliminate staging and focus more on being able to run experiments safely and efficiently in production.<p>in other contexts, the negative impacts of failure are very high. e.g. medical devices, mars landers, software governing large single systems (markets, industrial machinery). in these situations you might prefer to put more emphasis on QA before production.
What are some useful tools for running a development environment for each dev?<p>I have a pretty common setup of AWS services, Terraform, Docker, etc. Deploying this to a fresh AWS account is largely automatic but it takes about 20 minutes and it’s also expensive.
What works for you works for you. If you can't have a staging environment you obviously found a work around. There are many ways to deploy. Basically, you decide what risk you want to accept when you define a lifecycle.
I use my staging environment to let prospective clients or colleagues create and play with accounts without touching "real" data, and in the past used it to let a pentester test the non-prod site
>>>If we’re not confident that changes are ready to be in production, then we don’t merge them. This usually means we've written sufficient tests and have validated our changes in development.<p>This made me laugh.
This isn’t very uncommon. In fact, it actually is exactly what the article is trying to explain it’s not: a staging/pre-live environment. Only instead of having it be deployed online, you keep it local.
> Most companies are not prepared to pay for a staging environment identical to production<p>That's it basically no need for anything else. You are doubling, tripling ...etc your costs. No company will OK that.
How do people test in production? It's difficult to get test accounts in production because it containing consumer data. Think financial data.<p>Does anyone know any good resources for testing in production?
Without a staging environment, how do you test that large scale database migrations work as intended?<p>I wouldn't feel at all comfortable shipping changes like that which have only been tested on laptops.
Does QA just pull and test against a dev instance? Do they test against prod? Do engineers get prod API keys if they have to test an integration with a 3rd party?
Problem TL;DR:<p>"With staging:<p>- There could be differences from production<p>- Multiple people can't test at the same time<p>- Devs don't test their code."<p>Solution TL;DR: "Test your code, and push to production."<p>They completely misunderstood the problem and their solution literally changed nothing other than making devs test their code now. Staging could stay as is and would provide some significant risk mitigation with zero additional effort.<p>"Whenever we deploy changes, we monitor the situation continuously until we are certain there are no issues."<p>I'm sure customers would stay on the site, monitoring the situation too. Good luck with that strategy.
My experience with their list of suppositions:<p>> Pre-live environments are never at parity with production<p>My experience is that is is fairly trivial to have feature parity with production. Whatever you do for production, just do it again for staging. That's what it is meant to be.<p>> Most companies are not prepared to pay for a staging environment identical to production<p>Au contraire. All companies I've been to are more than willing to pay this. And secondly, it is pennies compared to production environment costs, because it isn't expected to handle any significant load. And, the article does mention being able to handle load as being one of the things that differ. I have not yet found the need to use changes to staging to verify load scaling capabilities.<p>> There’s always a queue<p>I don't undestand this paraph at all. It seems like an artificial problem created by how they handle repository changes, and has little to do with the purpose of a staging environment. It smells fishy to have local changes rely on a staging environment. The infrastructure I set up had a development environment be spun up and used for a development testing pipeline. Doesn't, and shouldn't need to rely on staging.<p>> Releases are too large<p>Well... one of the main benefits of having a staging environment is to safely do frequent small deployments. So this just seems like the exact wrong conclusion.<p>> Poor ownership of changes<p>This again, is not at all how I understand code should be shipped to a staging environment. "I’ve seen people merge, and then forget that their changes are on staging". What does this even mean? Surely, staging is only ever something that is deployed to from the latest release branch, which also surely comes from a main/master? The following "and now there are multiple sets of changes waiting to be released", also suggest some fundamental misunderstanding. *Releases* are what are meant to end up in staging. <Multiple set of changes> should be *a* release.<p>> People mistakenly let process replace accountability
> "By utilising a pre-production environment, you’re creating a situation where developers often merge code and “throw it over the fence"<p>Again. Staging environment isn't a place where you dump your shit. "Staging" is a place where releases are verified in an as much-as-possible-the-same-environment-as-production. So, again. This seems like entirely missing the point.<p>----<p>It seems to me that they don't use a staging environment, because they don't understand what such a thing should be used for. I'd be completely OK with someone rationalizing this as "too much of a hassle". But to try and justify something so poorly...<p>From their conclusion:<p>> Dropping your staging environment in favour of true continuous integration and deployment can create a different mindset for shipping software. When there is no buffer for changes before they go live, you need to be confident that your changes are fit for production. You also need to be alert and take full ownership of any changes you make.<p>Well... of course there is a shift in mindset when you'll be shitting your pants every time you make a change in production, since that's when you'll get to see if you broke something. The whole point of a staging environment is to have a buffer.... so that you don't have to be "confident". So that you don't have to be on high alert, because you have alerts that can trigger without anything important going offline. So that ownership isn't crucial in a post-fuck-up blame game.
I struggle with a lot of the arguments made here. I think one key thing is that staging can mean different things. In the authors case, they say "can’t merge your code because someone else is testing code on staging." It is important to differentiate between this type of staging for development testing development branches vs a staging where only what's already merged for for deployment is automatically deployed.<p>Many of the problems are organizational/infrastructure challenges, not inherent to staging environments/setups. Straightening out dev processes and investing in the infrastructure solves most of the challenges discussed.<p>Their points:<p>What's wrong with staging environments?<p>* "Pre-live environments are never at parity with production" - resolved with proper investment in infrastructure.<p>* "There’s always a queue [for staging]" - is staging the only place to test pre-production code? If you need a place to test code that isn't in master, consider investing in disposable staging environments or better infrastructure so your team has more confidence for what they merge.<p>* "Releases are too large" - reduced queues reduces deployment times. Manage releases so they're smaller.<p>* "Poor ownership of changes" Of course this happens with all that queued code. address earlier challenges and this will be massively mitigated. Once there, good mangers's job is to ensure this doesn't happen.<p>* "People mistakenly let process replace accountability" - this is a management problem.<p>Solving some of the above challenges with the right investments creates a virtuous cycle of improvements.<p>How we ship changes at Squeaky?<p>* "We only merge code that is ready to go live" - This is quite arbitrary. How do you define/ensure this?<p>* "We have a flat branching strategy" - Great. It then surprises me that they have so much queued code and such large releases. I find it surprising they say, "We always roll forward." I wonder how this impacts their recovery time.<p>* "High risk features are always feature flagged" - do low risk features never cause problems?<p>* "Hands-on deployments" - I'm not sure this is good practice. How much focus does it take away from your team? Would a hands-off deployment with high confidence pre-deploy, automated deployment, automated monitoring and alerting, while ensuring the team is available to respond and recover quickly?<p>* "Allows a subset of users to receive traffic from the new services while we validate" is fantastic. Surprised they don't break this into its own thing.
This sounds horrible unless they have a super reliable way to roll back changes to a consistent working state, both in their deployments and their databases.
Staging, tests, previews and even running code locally is for people who make mistakes. It's dumb and a total waste of time if you don't make any mistakes.<p>No testing at all, that's what I call optimizing for success!<p>On a more serious note:
Sometimes staging is the same as local, and in those situations there is very limited use for staging.
- I don't always test my code but when I do, it's in production.<p>- Everyone has a testing environment. Some people are lucky enough that they have a separate one for running production<p>[INSERT ADDITIONAL JOKES HERE]
> "We only merge code that is ready to go live"<p>I like to go even farther, I advocate only merging code that won't break anything. If you're feature flagging as many changes as possible then you can merge code that doesn't even work, as long as you can gate users away from it using feature flags. The sooner and more often you can integrate unfinished code (safely) into master the better.
Imagine writing this entire blog post and being completely wrong about every topic you discuss. This is the most amateur content I've seen make it to the front page, let alone top post.