| A couple of years ago, we experienced a silent data corruption incident in our checkout process due to this specific edge case. A user would generate the idempotency key by loading the front-end application, adding item(s) to their cart, submitting their order but timing out. The user would then navigate back to the front-end application and add another item and submit the order again. Since the user is submitting an identical idempotency key to the same transaction, our payment gateway would look up the request/transaction by idempotency key and see in its cache that there was a successful (200 OK) response to the previous request. The user now believes they purchased three items, however, our system only charged and shipped on two of the orders. Consequently, the lesson we take away from the aforementioned incident is idempotency keys are really composite keys (Client_Provided_Key + Hash(Request_Payload)). If a system receives an identical idempotency key (but with a different request payload) the idempotency key should be rejected with a 409 Conflict response with a message similar to "Idempotency key already used with different request payload". Alternatively, some teams argue it should be returned with a 400 Bad Request response. Systems should never return a failed cache response or replace old entries of data. This article explains how to unlock your flow. The final idempotent key will not be located until the first request completes, but will rather exist when the request is in progress. To safely accomplish your goal, you have to follow the following steps: 1. Acquire a distributed lock on the idempotent key. 2. Check for the existence of a key in your persistent store. 3. If an existing key is found, verify the hash of the payload against the hash for the payload type. If the hashes do not match, return a 409 error. 4. If the hashes match, look up the status of the payload. If the status shows COMPLETED in the persistent store, return the cached response. If the status shows PENDING in the persistent store, return a 429 Too Many Requests to the user or hold the connection open until the request reaches a PENDING state. 5. After processing the request, save the response to the persistent store before releasing the lock. While this may look simple on paper, creating a distributed locking state machine for a single API endpoint is typically how developers have their first aha moments with idempotency. Becoming idempotent is often an enormous architectural shift and not just a middleware header check. |
An API should follow its documented behavior. This is both a specification and a contract. If the docs for the API say that a duplicate idempotency key will receive a 409, and do not mention message hashes, then they need to follow that spec because the client may specifically depend on it. For example if the order was processed and the cart is resent with the same key but an additional item, client does not want another order with the duplicate items in the first one. They want an error.
If the docs do not accurately describe the behavior of the idempotency key, the client should find another provider.
> While this may look simple on paper, creating a distributed locking state machine for a single API endpoint is typically how developers have their first aha moments with idempotency. Becoming idempotent is often an enormous architectural shift and not just a middleware header check.
Yes, when you expand the scope of your API implementation beyond its contract you take on a virtually unbounded amount of edge cases that not only must you solve, but that your customers must guess at how you are solving.
I'm guessing that your API required the idempotency key. I think that is could be risky because it means developers will simply provide a value for it without understanding the purpose, or thinking through the implications. You really only want them using it if they understand the problem it is solving.
Hashing message content could be an alternative behavior that it makes sense to support by default for apps that don't supply an idempotency key. As long as you document it.