Kafka has shipped the long-awaited ZooKeeper-free mode, but AFAIK it’s still beta and behind feature flags on the producer, broker, and consumer (like almost all Kafka config :( but that’s another story)
Yeah, it's shipped, but it's missing some existing ZK features that tooling around Kafka relied on, and I'm a bit embarrassed for Confluent that they pushed KRaft so hard without a replacement.
E.g. the ability to watch a ZK node for changes, which means in Kafka sans ZK, you can't detect changes to topics without continuously polling via the admin client.
A coworker is working to implement something like this for KRaft, but it really demonstrates how an IPO can cause a company that was the steward of a FOSS project to do things detrimental to that project to keep the share price up. (Was also interesting how many key Confluent people left right after the IPO)
The other very notable change is how Confluent's dev effort has switched from the open source project to the Enterprise Edition, but they still have the majority of PMC members, while not having the corporate blessing to spend time reviewing PRs.
Sorry, realised I'd lapsed into jargon and missed the edit.
KIP = Kafka Improvement Proposal. The mechanism for proposing big changes to Kafka and getting community feedback before core committers vote on adoption or not.
I'm not sure why Zookeeper is viewed so negatively in regards to Kafka, it's damn solid, and if I can quote Jepsen "Use Zookeeper". I know Confluent wanted to replace it because it struggled when you hit thousands of topics on a single cluster, but that feels like a very niche use case to me.
I agree re: zookeeper. It’s rarely been the part of the stack making me lose sleep. ~~Raft~~ seems like a way to “modernize” Kafka by detaching it from the Hadoop ecosystem — and I think that’s about it. :(
E.g. the ability to watch a ZK node for changes, which means in Kafka sans ZK, you can't detect changes to topics without continuously polling via the admin client.
A coworker is working to implement something like this for KRaft, but it really demonstrates how an IPO can cause a company that was the steward of a FOSS project to do things detrimental to that project to keep the share price up. (Was also interesting how many key Confluent people left right after the IPO)
The other very notable change is how Confluent's dev effort has switched from the open source project to the Enterprise Edition, but they still have the majority of PMC members, while not having the corporate blessing to spend time reviewing PRs.