[jrochkind1]

If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm.

i may be missing something though?

[luuio]

let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed.

when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests?

you're mixing up conflict handling vs. idempotency

[jrochkind1]

Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system?

Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!

[luuio]

Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation.

Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded

2. The database returned an acknowledgement, but then the callsite crashed before storing the acknowlwedgement for the next step.

[jrochkind1]

You are under no obligation to continue this conversation of course.

> Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation.

But how do people use, say, redis, without an external "checkpoint" system, how do you do, say, an INCR operation, and know if it succeeded or not?

Or are most uses of redis actually dangerous and subject to these error conditions, perhaps it's tolerable to risk that error for most redis use cases, but wasn't for the system under discussion? Most developers using redis definitely aren't using external "checkpointing" systems, or considering if they should be or not -- should they be?

But of course you've convinced me that the system under discussion would have been better off using an rdbms, something I never doubted.

[luuio]

Happy to continue :)

Where our conversation is at, it's not about RDBMS vs. NoSQL.

On the Redis INCR operation example (or an equivalent operation in any database, even in an RDBMS). In short, you don't always have the guarantee to know the result of the operation. This is part of what the 99.9% means when services advertise their reliability. 99.9% of the time you'll get an acknowledgement back, but not 100%.

The question comes to, what happens if I miss an INCR, or if I accidentally double INCR when I retry? For some use cases, it's acceptable: like counting the number of views on a Youtube video. For others, say a financial system counting money or making a payment, that's not OK. That's where idempotency really matters (btw this is why many redis operations have the NX option).

Let's go completely off track for a little bit, say you're Strava. You let devices and users upload workouts to your service. What happens when a phone sent you the workout, then disconnected due to bad receptions? In this case, the phone doesn't know if the operation succeeded so it's forced to retry. How do we make sure the workout isn't created and counted twice on Strava? Well, each workout has a unique UUID generated by the device. Strava uses this UUID to dedup when it receives an upload. The UUID is the idempotency "checkpoint".

In this case, it doesn't matter what kind of database Strava is using. The idempotency problem is the same.