[hardwaresofton]

Has anyone used both Zookeeper and etcd in production for management of distributed state?

Generally when I think of this problem I reach for etcd, not Zookeeper first, in the hopes of it being lighter (with a relatively vague definition of "light"), and easier to use.

[neeleshs]

Not etcd, but zookeeper yes. It's mostly been set up and forget kind of infrastructure for us, except for some snafus created by our own config errors.

[bkeroack]

etcd would be a fine choice, but Consul provides a lock feature[1] out of the box.

1. https://www.consul.io/docs/guides/leader-election.html

[teraflop]

As the article explains at length, using Zookeeper or etcd as a "locking service" in this way is not safe, even if the service is perfect and failure-free. There is a fundamental race condition between finding out that you've obtained a lock, and doing some other operation that assumes mutual exclusion.

[bkeroack]

It's not really "fundamental". It's simply that the process that acquires the lock can fail (or be paused, or partitioned away from everything else, etc), and if it does, but then comes back later with a valid lock, bad things may happen.

The author's solution is to push serialization logic into the resource/storage layer (by checking fencing tokens). But what if the resource is itself distributed? Then it needs it's own synchronization mechanism? It's locks all the way down.

[bkeroack]

Thinking more about it, this is a fundamental weakness of having self-policing processes, which I suppose is the OP's main point. It can be mitigated by having infinite lock TTLs, at the cost of risking system deadlock on process failure. Thank you to GPP for spurring me to think more deeply about this.

As I stated, though, if the resource being protected is either a distributed system itself, or a system that cannot support fencing logic, this failure mode is difficult or impossible to prevent. The frequency of failure should be kept in mind here: most services can probably guarantee 99.99% uptime against the likelihood of 5 minute GC pauses.

[takeda]

Doesn't it have the same issue that redis does?

[harlowja]

Nope, zookeeper nodes have ids that always increment and can be used for fencing, compare-and-set and more...

[takeda]

I was talking about consul and etcd.

[syscomet]

In the linked consul documentation, we see a `LockIndex` field in the lock state; https://www.consul.io/docs/agent/http/kv.html confirms that "LockIndex is the number of times this key has successfully been acquired in a lock. If the lock is held, the Session key provides the session that owns the lock."

So you can do fencing with consul.

[syscomet]

For etcd: https://coreos.com/etcd/docs/0.4.7/etcd-api/

node.modifiedIndex appears sensible for fencing.