[hugofirth]

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story.

On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but it also means that more people are familiar with the quirks, limitations, requirements etc.... of these tools. For example, I think I would be frustrated to hear that K8s were implementing their own raft.

[sagichmal]

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

[saurik]

I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

[skyde]

100 % agree we need good (etch/zookeeper) as a library.

I wrote something similar but that provide only (lock/lease) using paxos.

The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...

[lima]

The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.

[skyde]

yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API.

This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.

[chucky_z]

Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?

[sagichmal]

Sounds good to me, more things like ZooKeeper should be embeddable.

[halbritt]

The fact that many people are scared away from Kafka simply because they don't want to run zookeeper should be telling.

[manigandham]

This is the difference between a library and a framework/project.

For example, there's a great Raft library for Go [1] that any project can use to implement distributed consensus without a separate running program. I find this to be a better approach with the same collective development and community testing advantages but without requiring more operational overhead.

1. https://github.com/hashicorp/raft

[skyde]

this lib u is great and 1000 time cleaner better tested and easier to use compared to trying to reuse etcd raft implementation :-)

[smarterclayton]

The difference is that Kubernetes isn’t a data store - we’d have to implement the full functionality of etcd. Kafka is a full featured data store that doesn’t need 80% of what zookeeper does.

This was actually an early complaint leveled against Kubernetes for things like proxying at the node level or implementing DNS. “Don’t reinvent the wheel!” Sometimes better administrative experiences exist only after a component absorbs some function previous systems expose.

Sometimes it’s better to own the parts of the problem that make your system simpler.

[justicezyx]

No etcd don't need to be replicated for k8s API server.

Basically versioning + some form of fault tolerance is sufficient for k8s API server.

[kirstenbirgit]

I generally think it makes sense to start with readily available tools, but as a project grows and becomes more specialized, it often shows that the tool doesn't quite fit, doesn't quite have the same performance characteristics, or otherwise not a good fit anymore. And then it's time for something better. Or, it works out, and you don't switch it out (e.g. etcd in k8s)

[gregwebs]

There are a lot of shared raft libraries now. It is much easier to share library code than a piece of a distributed system.

[whorleater]

If ZK wasn't such a pain in the ass to manage I'd agree with you.

[skyde]

this pain come from paxos raft or Zab algorithm used by concensus system. I don’t think it can be avoided.