Hacker News new | ask | show | jobs
by SamBam 2336 days ago
No? They just described QA testers or actual users finding the bugs.

i.e. the change made it through the tests, and now is deployed live, where a human may encounter and report it.

1 comments

That's applicable to a toy project.

If it is not a toy project then QA testing does not scale because there are thousands if not millions paths that request can flow through the infrastructure based on a user's actions.

That's why it is imperative to augment and nearly replace QA with end-to-end integration tests. Not only that but the integration tests must be funneled though the production gates to have a reasonable assurance that the code works in production configuration -- such as production request/task budgets, production latency and production tracking/tracing.

That in turn requires the test code being gated from production traffic ( which requires infrastructure to support request pinning ) and monitored/instrumented ( requires request tracing ) and actions ( error rate is 77 reqs/second when the test generates 4 req/sec means that the non-test traffic is landing on the test - remove test deploy immediately ).

So we are back to the integration tests nearly a 100% coverage to allow for unattended deploy.