[justin_vanw]

What is the negative consequence of breaking the site for an hour? Is it "some people are a little annoyed", "nobody notices", or "customers start calling and are very upset" or even "customers file lawsuits"?

The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.

[websitescenes]

Having a staging environment is pretty standard these days. We set one up from day one at our startup. It doesn't really take much effort and your app benefits from being decoupled from the domain. This allows you to run app instances concurrently without relying on domain specific details. It also allows you to physically test changes before pushing to production.

I can hardly see how a staging environment is "recommending big company super risk averse solutions at a startup, which is just silly". A staging environment has never slowed us down; if anything it gives us the confidence to push to master sooner.

[stestagg]

I think most of the positive statements across this thread are generally correct, and most of the negative statements show a lack of awareness/inexperience:

My experiences are:

1. Staging environments are typically imperfect and have issues

2. Staging environments often help with product stability despite (1.)

3. Having staging envs is common, and could be considered standard in many companies, but..

4. Staging environments are a cost-benefit trade-off and should be considered on a product-by-product case.

5. CI and Unit testing are also common approaches to increasing product stability, and should be subjected to the same cost-benefit analysis per-product (yes, even unit tests), It's not an either-or with staging environments

6. staging envs should be prod-1 but often they end up being (prod-1 + some workarounds + some cruft left over from a failed deployment + other noise) over time.

7. Running a fully prod-equivalent staging environment can be prohibitively expensive (exact replica of prod configuration including infrastructure, realistic copies of prod data at volume, etc..) so usually some compromises are made to give a best-effort simulation of prod (fewer resources, etc..).

8. 'Battle testing' in staging is not always done, it's not uncommon for staging to be largely not looked at, except for whatever automated testing/alerting is running in that environment

9. It's ok for staging environments to test multiple changes at once, the typical intent of a staging area is to identify the presence of problems before they hit prod. The debugging & isolation of what caused problems is a secondary issue.

Your experiences may differ from the above, but they still stand

[edit: formatting]

[websitescenes]

Good points. Your description is largely similar to my own experiences. On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken. There is not a single piece of staging specific code in our app. Whatever works on staging works on production. This is accomplished with ENV variables instead of hard coded environment specific details. We actually push directly from staging into production.

[hluska]

> On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken.

I'm not the person you're replying to, but in my experience, this only happens if you get 100% buy in to the dev ops approach. If you get less than 100% buy in, you'll often end up with greasy environments with the extreme end being a founder type writing code directly in prod.

[dfee]

> A site like twitter can go down and slightly improve the life of its users.

Ha!

[whack]

> Besides, staging environments never work, for a long list of reasons

Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.

[bjourne]

But in that case you have qualified people actually performing battle-testing. If you don't, then a staging area can be more than useless because developers think that making sure their code is bug free is someone else job. Also in most companies testing is deemed inferior to programming, so the least capable is assigned testing duties while the better programmers write the code. Which is wrong because battle-testing software takes just as much skill as developing it.

[Xeago]

Unless you can spin up and tear down a staging environment provisioned with data and instrumentation I too don’t believe in it. Without that you’ll be using something shared, where either one thing at a time is subjected to testing or you’re testing multiple things at once which could conflict (including falsely positive) with eachother.

Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.

[Chyzwar]

Staging env should be (prod - 1) version. The whole point is to test thing together with other changes. Stage-environment is created to eliminated "works on my machine" problems.