[Acconut]

At my workplace, we are currently looking for a distributed locking mechanism. Since we are already heavily using Redis in our system, it seems to be convenient to rely on it, too, but we heart a lot of criticism against it.

Does anyone have experience with distributed locks and would be kind enough to share?

[craigjb]

In a real-time system at work, we originally used Redis as a locking mechanism to avoid resource contention. However, it turned out the better long-term solution was to better use our reliable message broker and make tasks finer grain.

So, rather than lock large resources, we would break the processing into tiny tasks, each completely contained in a message object. Each message specified a new message to launch on completion. This way the messaging protocol is preventing resource contention rather than a lock.

[edit] The context involved some licenses software to which we only owned several instances. The original approach would attempt to grab a lock and run the batch. The final approach just created a queue for jobs and one consumer for each available license. Now, none of our compute is spent waiting on locks either.

[Acconut]

Splitting a resource into multiple parts and embedding in into a single message in order to avoid external storage and the need for locking is an interesting approach, but sadly not an option for me. Nevertheless, I appreciate your response.

[craigjb]

Yah, I understand it's not always an option.

If the data size is limiting though, you can embed a Redis key or Hadoop FS reference into the message, rather than breaking your message system with massive data.

Then, you can have the storage system listen for completion messages, and cleanup appropriately. Or, like we did, just shove out old data LRU style. This way some history is always available for debugging.