[atombender]

Redpanda [1] is fantastic if you want a solid Kafka clone. It's written in C++, with no external dependencies; it has Raft consensus built in. The API is Kafka-compatible.

There are some missing features compared to Kafka (no dynamic rebalancing of partitions being the top one), but they're also rolling out some new features that Kafka does not have, such as transformation pipelines [2].

NATS's brand new JetStream [3] feature is also looking very promising. It uses Raft internally. NATS itself is rock solid, and JetStream builds on that foundation.

NATS also has very interesting support replication topologies, meaning you can build graphs of streams that each feed in other streams, so you can do the publishing and consumption in different locations, with different availability constraints.

[1] https://vectorized.io/redpanda

[2] https://vectorized.io/blog/wasm-architecture/

[3] https://docs.nats.io/jetstream/jetstream

[kevstev]

Do you have actual production experience with RedPanda? If so I would love to hear about it- we found out that there is something of a wall of 10k partitions per broker before things start failing/under replicating without warning or any issues outside of the URPs. This appears to be a limitation of zookeeper and fetching metadata. We are fighting this by raising timeouts and such, but this blindsided us, and really the solution is to get people to stop creating topics with dozens of partitions when they aren't needed.

I took a look at redpanda this week, it sounded nice on paper, but them being a young company, I am concerned about what "gotchas" we are going to run into.

[atombender]

I've only run it as part of stress tests when evaluating it for a new application, and I found it to be a pleasure to work with.

We did push it to more than 10k partitions, but I honestly don't remember how that affected it; that's when I discovered that partitions cannot be dynamically rebalanced, which meant we'd have to change the way we would use it.

Interestingly, we also did a similar test with NATS JetStream, which did start struggling around 10,000 consumers. (A consumer in NATS is similar to a partition, as it has its own Raft group.) What I tried to do with JetStream goes against the grain a bit, mind you; I still think it's an excellent piece of software.

[derekcollison]

We have some great work coming that allows lighter weight consumers to scale to that level and beyond. Happy to chat with folks on how we can make that work today.