[tveita]

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka.

(That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

[rad_gruchalski]

> Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka.

BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.

[EdwardDiego]

It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO.

And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.

[rad_gruchalski]

> offload old data into S3 and then retrieve it if required

I remember asking for it 5 years ago: https://radek-gruchalski.medium.com/the-case-for-kafka-cold-.... Confluent turned it into a paid feature.

[EdwardDiego]

Yep, it's available from Confluent if you pay for it, but like how Mirror Maker 2 is awfully similar to Confluent Replicator, I believe that Kafka will (eventually) get tiered storage under the Apache licence (I know there's a KIP for it[1]). It's a hard issue to solve, and not sure how much effort in the community is being directed towards it. But bear in mind that it's not just Confluent who have a stake in Kafka - there's a bunch of big corps selling managed/supported Kafka and all of them would probably quite like tiered storage in core Kafka as a feature to help them sell their support/management, so I have some faith in their enlightened self-interest.

The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith.

Incidentally, I quite like how Pulsar solved tiered storage, and it's a definite tick in the Pulsar box - it's transparent from a consumer's POV, although there somewhat of a delay in rehydrating the offloaded block, I don't think anyone's expecting near-realtime performance when loading historical data.

[1]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...

[miguno]

Thanks for putting things in perspective, EdwardDiego.

> The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith.

Let me share an anecdote related to this example. We (Confluent) were actually the ones who contributed the documentation for MirrorMaker v2 to the Apache Kafka docs (https://kafka.apache.org/documentation/#georeplication). The development lead on MM2 was (an engineer at) Cloudera, yet they never spent the time to provide user-facing documentation to the Kafka project. I don't want to speculate about reasons, yet I noticed that MM2 was documented in the Cloudera docs.

If we didn't care for the Kafka community at Confluent, we would not have spent our own resources and time to fill that gap, given that we have a proprietary product similar to MM2 (i.e., Confluent Replicator).

https://github.com/apache/kafka/pull/9983

[EdwardDiego]

Shit, wait, there's documentaton for Mirror Maker 2 now? I spent most of my time implementing it by reading hypothetical examples in a KIP, and then diving into the actual code.

Hardly the most straightforward, and it was rather a gaping hole. Thanks for the background on how that hole developed.

I really appreciate Confluent putting that time into documenting something vital, that could compete with your own product, and IMO that does put a nail in the previous commenter's assertions about Confluent's alleged attempts to wall off necessary features of Kafka.

[skyde]

I think you are completely missing the point. Those kind of system need to be designed in layer. You can host all 3 layer on each of your VM instance if you want but they should not be mixed in the same process.

One layer BookKeeper provide an abstraction similar to HDFS. That is it provide file that are horizontally scalable in size and throughput and reliable append only files.

Pulsar is a service built on top of BookKeeper but could run on top of HDFS or something like Amazon S3 ...

And is only responsible for making sure there is only one writer per BookKeeper file even if multiple process try sending request to Pulsar to write to the same partition. It also try to balance request across all the brokers.