[sargun]

Funnily enough, this is actually a massive problem when working with cloud automation APIs. Terraform and the like kinda handle this problem by calculating / storing the “goal state” and then looking at the system’s current state, and coming up with a “plan” to reconcile it.

Unfortunately, cloud provider APIs are usually eventually consistent, and getting a full snapshot at scale is nigh impossible.

So, in order to work around this, I effectively built a write ahead log style atop Postgres. Something like Sagas would have been great, but as far as I can tell, there was no real pattern for multiple Sagas operating on global state doing coordination. This is where Postgres SSI came in handy, where I could read the assumed state of the system, and if another worker came in and manipulated it, the write ahead entry wouldn’t get written as the txn would fail to commit. The write ahead entry would then get asynchronously processed by another worker, in case the first worker failed.