[ovis]

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions.

2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

[rad_gruchalski]

Re 2, have you ever tried using Kafka with a non-JVM client?

[EdwardDiego]

Yep, I have, no issues with the consumer/producer knowing who the partition leader is.

That said, curious to hear your experiences :)

[rad_gruchalski]

I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up.

Outside of librdkafka and jvm client, it’s gloves off.

[EdwardDiego]

Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions.

IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going.

That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients.

Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.

[mrkeen]

kafkacat is excellent.

https://github.com/edenhill/kafkacat

[rad_gruchalski]

Definitely, if all one needs is command line.