Hacker News new | ask | show | jobs
by cmckn 1437 days ago
> you can't detect changes to topics without continuously polling via the admin client.

Yikes, that sounds like an oversight! Aren't topic configs written to a system topic that you could consume from?
1 comments
EdwardDiego 1436 days ago
They were supposed to be, in the original KIP, but that changed, I'm unclear as to what drove that change.

So my coworker's solution joins the KRaft quorum as an observer, then publishes metadata changes to a topic you can consume from.

link
EdwardDiego 1436 days ago
Sorry, realised I'd lapsed into jargon and missed the edit.

KIP = Kafka Improvement Proposal. The mechanism for proposing big changes to Kafka and getting community feedback before core committers vote on adoption or not.

I'm not sure why Zookeeper is viewed so negatively in regards to Kafka, it's damn solid, and if I can quote Jepsen "Use Zookeeper". I know Confluent wanted to replace it because it struggled when you hit thousands of topics on a single cluster, but that feels like a very niche use case to me.

link
cmckn 1433 days ago
That sounds like a handy project!

I agree re: zookeeper. It’s rarely been the part of the stack making me lose sleep. ~~Raft~~ seems like a way to “modernize” Kafka by detaching it from the Hadoop ecosystem — and I think that’s about it. :(

link