[jpdb]

There seems to be a misunderstanding regarding what Continuous Delivery (CD) is, as well as what is all required.

CD doesn't necessarily refer to Deployment of Delivered artifacts. Instead, once code is merged back into your release branch (sometimes master branch, sometimes tag, sometimes an actual release branch), it should be made available to be launched to production. This doesn't mean more testing isn't required and it doesn't mean that humans are not involved. Continuous Deployment is the process of releasing software to production automatically.

If you did in fact mean Continuous Deployment then you do not need 100% integration test coverage or anything of that nature. Typically, the process might work by releasing a canary deploy which receives a fraction of the traffic going to production. In the unlikely event a breaking change has silently made it through your other tests, it should likely be caught when it is deployed as a canary. If in the extremely unlikely scenario it makes it through both then the existing pipeline you have can be used to "roll-forward" a fix automatically instead of via the human process that is much more likely to fail.

This may seem like a lot to you, but you are front-loading quite a bit of work and a mature pipeline is much less error prone than manual deploys.

[choeger]

Note that "canary deployment" (I hate the usage of 'deploy' as noun.) is only possible under specific circumstances. You would not want to do that with your database, for instance (aka, "0.5% of your data might now be corrupt or lost) or your development tooling (aka, 0.5% of your deployed code might not do what it should do according to its source code).

[plopz]

You also need a large enough user base/level of traffic where releasing to a small percentage of your users would even uncover a problem.

I've had similar problems with A/B testing where a low amount of traffic means I would have to wait a long time to get meaningful data during which confounding variables naturally get introduced.

[notyourday]

> If you did in fact mean Continuous Deployment then you do not need 100% integration test coverage or anything of that nature. Typically, the process might work by releasing a canary deploy which receives a fraction of the traffic going to production. In the unlikely event a breaking change has silently made it through your other tests, it should likely be caught when it is deployed as a canary

You have just described 100% integration test coverage.

[SamBam]

No? They just described QA testers or actual users finding the bugs.

i.e. the change made it through the tests, and now is deployed live, where a human may encounter and report it.

[notyourday]

That's applicable to a toy project.

If it is not a toy project then QA testing does not scale because there are thousands if not millions paths that request can flow through the infrastructure based on a user's actions.

That's why it is imperative to augment and nearly replace QA with end-to-end integration tests. Not only that but the integration tests must be funneled though the production gates to have a reasonable assurance that the code works in production configuration -- such as production request/task budgets, production latency and production tracking/tracing.

That in turn requires the test code being gated from production traffic ( which requires infrastructure to support request pinning ) and monitored/instrumented ( requires request tracing ) and actions ( error rate is 77 reqs/second when the test generates 4 req/sec means that the non-test traffic is landing on the test - remove test deploy immediately ).

So we are back to the integration tests nearly a 100% coverage to allow for unattended deploy.