Hacker News new | ask | show | jobs
by schmichael 93 days ago
As a longtime Raft user (via hashicorp/raft), I'm curious about your Raft implementation! You mention etcd's Raft library, but it isn't natively Multi-Raft is it? Is your implementation similar to https://tikv.org/deep-dive/scalability/multi-raft/ ? I'd love to hear about your experience implementing and testing it!
1 comments
carpenterant91 93 days ago
Awesome question! We'd experimented with https://github.com/lni/dragonboat and the hashicorp/raft in the early implementation, the etcd/raft library had been ported to a multi-raft style implementation by CockroachDB way back when, but they went the way of TigerBeetle and coupled their consensus deeply with the kv storage. Etcd has recently in v3.6 abstracted out their raft implementation and gave a pluggable interface into the transport layer, which meant that we could implement our own multi-raft style transport layer with heartbeat and multi-node message buffering on top of HTTP/Quic.

We implemented chaos testing suites akin to Jepsen to cover as many scenarios as possible and are currently implementing TigerBeetle style simulation tests on top of that for harder to reproduce scenarios!

link
schmichael 93 days ago
HTTP/QUIC, so no gRPC then? Or is https://github.com/grpc/grpc/issues/19126 not the blocker to gRPC over QUIC I thought it was.

I've long wished for QUIC with Nomad! [1] We've always used a weird QUIC-over-TCP multiplexer called yamux. [2]

[1] https://github.com/hashicorp/nomad/issues/23848

[2] https://github.com/hashicorp/yamux (I'm fairly certain libp2p's fork is actually better)

link
kingcauchy 93 days ago
Fascinating! We settled on Quic with Protobuf because it was more performant in our testing than the gRPC when coupled with the backoff, failure cases (node startup ordering server/client connections), and to not be coupled with the gRPC library versions in Go, which has bitten us a number of times when dealing with dependency management when you're trying to juggle k8s, etcd, and google dependencies in the same Go project. Plus the performance bottleneck in most of the use cases we're specializing in are on the embedding/ml side of things.

Thanks for the links! I hadn't seem yamux before!

link