[krnaveen14]

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs.

- Decoupling of Broker & Storage Layer

- Tierered Storage (SSD, HDD, S3,...)

We use both Kafka and Pulsar in our systems.

- Kafka is used for microservices communication and operational data sharing

- Pulsar is used for streaming large customer data in thousands of topics

[lizthegrey]

In 2016 when we were founded, Pulsar wasn't a thing yet. Today, we'd be very likely to use Pulsar if we were starting from scratch.

[2p_locking]

Agree. We did a recent migration from Kafka (Confluent Platform) to OSS Pulsar for a near identical use case and couldn't have been happier - we've saved costs, improved uptime, and reduced errors. We found it much easier to manage Pulsar ourselves with automation that it mostly removed the need to use a managed Kafka service (we've tried solutions from AWS, Confluent, and Aiven).

The only downside with Pulsar (which is constantly improving) was that community support for teething issues was a bit slower than what we saw with Kafka.

[toomanybits]

Now Kafka has ZK taken out and tiered storage coming in these factors are becoming mute points. With tiered storage Kafka has 2 layers and hence the benefits of broker/storage decoupling. By comparison, Pulsar has broker, bookkeeper, S3, and ZK layers to contend with. This is why a second layer was never added to Kafka directly.

[staticassertion]

What's the story like for migrating from Kafka to Pulsar?

[krnaveen14]

- Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times...

- Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics)

- Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale

- Tiered storage was not available previously in Kafka (is it now available in open source version?)

- Native Multi Tenant support with Authentication / Authorization support not available in Kafka (essential for customer based thousands of namespaces and topics)

[helsinki]

> Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times...

It also uses num_partitions*2 open file descriptors per topic, which can quickly surpass the default ulimit on a host. Always remember to raise the ulimit before you near 1000 topics, otherwise Kafka crashes.

[lizthegrey]

right -- the question I and others have is, can you convert an existing topic in place or do you have to do the dual-writer/shadow launch, then drain the old one model etc.

[krnaveen14]

We choose both Kafka and Pulsar from the start. Haven't done any migrations from one to other.

If we are gonna move from Kafka to Pulsar (operational data),

- Publish new messages only to Pulsar

- Wait for Kafka Subscriber to process all pending messages

- Start Pulsar Subscriber (step 2 and 3 can done in parallel or sequential based on message ordering needs)

We don't need to worry about pulsar subscriber offset since existing messages in kafka topic won't be copied to pulsar topic preventing duplicate messages. The whole migration would be completed in 1-2 min (depending on Step 2 wait time).

This approach is carefully taken such that there can be little lag in subscriber but the publisher should not be affected by any means.

[BiteCode_dev]

What's the difference between Kafka, Pulsar, RabbitMQ and Crossbar.io ? They all seem to be hubs for PUB/SUB and RPC.

[polskibus]

Is Pulsar a good fit for microservices communication ?

[krnaveen14]

Yes, definitely. It's just that we used Kafka in the start for microservices communication and later choosen Pulsar for large data streaming.

We have it in our roadmap to migrate the former to Pulsar, but not the utmost priority right now. It also simplifies Infrastructure management by keeping one event streaming stack instead of two.