|
|
|
|
|
|
by mjb
4179 days ago
|
|
|
I'm the same 'mjb' from the linked lobste.rs thread. If anybody is interested I expanded that comment into a blog post here: http://brooker.co.za/blog/2014/11/15/exactly-once.html The core problem here is making the act of writing down that you've done something atomic with doing the actual thing. If you can solve that problem, exactly-once processing is easy. If you can ignore that problem, for example because doing the thing is idempotent, exactly-once processing is easy. In the real world, however, it can be really difficult to solve these problems in general and very specific (and often 'incorrect') solutions are used. OP's post talks about a bunch of those, which is very useful. Very often, though, real-world systems settle for 'at least once' or 'at most once' and find out-of-band ways to handle the missing or duplicate messages. Whether this is practical or not depends on the message rate, and the cost of getting it wrong. |
|
|
I enjoyed and totally endorse the content of your blog post, but I did find the title / intro a bit misleading. Exactly-once is pretty much always what people want: it's the easiest model for most folks to reason about, especially when they're new to the whole distributed-systems thing. (If exactly-once delivery was easy, you certainly wouldn't find people going out of their way to build systems that dropped or duplicated messages.) Of course, you might not need exactly-once delivery to get what you want -- which is good, because like you say, that's a hard / impossible problem to solve in general.
It feels to me like people in the stream processing universe have taken that message a bit too much to heart -- things like the 'lambda architecture' treat stream processing as "too hard to get right" and relegate it to approximate, disposable calculations only. My post was partly an effort to push back on this; as a community, I don't think we should give up without a fight.