[math]

There is currently (much anticipated!) work underway to remove Kafka's Zookeeper dependency: : https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

As of v0.9 (about 4 years ago), clients have no use for a direct connection to Zookeeper, and don't maintain one.

Fat clients are more effort to make, which is a significant downside, but this is abstracted from the user and Kafka has critical mass - the client story outside Java is improving dramatically. A fat client allows for optimal availability, throughput, latency.

[atombender]

Thanks, that's promising.

A fat client makes a lot of sense, but if it's written in Java, then only JVM-based languages benefit. A client written in C, C++ or Rust — or any other language with a similarly good C-interop story — could easily be consumed by just about anything except, ironically, Go.

[math]

there are pros and cons to having a C dependency. the main benefit is reliability (get it right once, leverage that across languages). Another benefit is usually performance. The main practical gotcha is that security dependencies are incompatible across platforms, and the best you can do is distribute a number of builds of the C dependency compatible with common platforms (but that is workable).

the go story isn't as bad as you might think. there is a performance hit with the interop, which is of the order of 20% iirc - but throughput still massive, and this is not of practical significance. The Confluent go client (a binding to librdkafka), is used to push much of the volume through two of the largest Kafka installations in existence.

there is a PR open on the confluent go client that will include builds of librdkafka in the github repo, meaning this will 'just work', without the need to install librdkafka separately in most common scenarios.

note: i work on kafka clients (including minor contributions to the confluent go client).

[atombender]

Thanks -- I thought the Go stack switch overhead was more than 20%, that's not awful. A lot better than JNI, I think?