[sagichmal]

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

[saurik]

I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

[skyde]

100 % agree we need good (etch/zookeeper) as a library.

I wrote something similar but that provide only (lock/lease) using paxos.

The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...

[lima]

The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.

[skyde]

yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API.

This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.

[chucky_z]

Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?

[sagichmal]

Sounds good to me, more things like ZooKeeper should be embeddable.

[halbritt]

The fact that many people are scared away from Kafka simply because they don't want to run zookeeper should be telling.