[yencabulator]

This sort of architecture often shows up in actors, e.g. https://www.microsoft.com/en-us/research/publication/orleans...

In that world, what you're generally looking at is

local state + incoming message -> new state + outgoing messages

Outgoing messages are sent only after persisting, and will be retried until success. Unique message IDs are used for idempotency (also for incoming messages).

Important: the actor runs with no side effects -- that's what makes rerunning things safe. In that kind of world, side effects are often achieved via e.g. sending a message that updates a materialized view somewhere (see e.g. Kafka).

With this setup, the source of badness is often isolated to the incoming message, and after a few failures an incoming message can be moved into a "dead letter queue" for ops to look at. In many scenarios, this actually works remarkably well.

https://pmatseykanets.github.io/beanstalkd-docs/protocol/#bu...