[sethammons]

In unit tests, we use configurable timeouts and/or mock time (dependency injection usually). This lets us keep tests fast.

In integration tests, we black box test our service from the edges. Docker Compose ftw. We validate logs and other output such as metric endpoints. If we read from a queue, we run that queue; same with all our dependencies. Sometimes we will use fake services with mock/default responses if running the dependency is too onerous.

System testing. We run a smaller clone of production as a staging environment. Automated tests kick off customer actions (say, using an API or submitting a file for processing) and monitors logs and sinks (customer webhooks, user notification, emails, files generated, etc). These test suites exercise the entire system from the customer point of view.

Production. Metrics, metrics, and logs (and alerts). Deployment dashboards of metrics and error logs, slo/sli graphs, and automatic rollback on some metric violations. We usually canary a single node, the do a multi-tier roll out.

When a bug shows up, we determine which level of testing makes the most sense and add a test case to prevent future regression. New alerts may be set up.