Ask HN: How do you ensure that a distributed app is working as expected?

[aprao]

Imagine a very simple user creation flow in an online marketplace:

- Service A (user service) receives the request and creates a user object and sends an async request to service B and C (e.g. via Kafka)

- Service B (notification service) receives the request and sends an email to the newly created user

- Service C (referral service) receives the request and credits some funds to the referrer

While this design might be laid out correctly in a design doc, it is only implicitly defined in code because the services talk to each other. How would you:

- Ensure that the services are talking to each other in the correct order when implementing the user creation flow (integration tests might not suffice here since they generally test a very narrow set of path)?

- Define and enforce SLO guarantees between services in production?

- Debug which service is to blame when the flow breaks down?