[hosh]

This isn't a tutorial, but this paper has been blowing me away: http://www.neilconway.org/docs/socc2012_bloom_lattices.pdf

I bet analyzing those messages using this framework would reveal exactly why compensating actions are needed.

The takeaway I am getting from this paper is that, you're not just looking for idempotency. Monotonicity is important, as well as commutativity and associativity. If it cannot be expressed in that way, then coordination is required.