[EdwardDiego]

Why though?

It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka.

https://blog.twitter.com/engineering/en_us/topics/insights/2...

> One catch to this is that for extremely bandwidth-heavy workloads (very high fanout-reads), EventBus theoretically might be more efficient since we can scale out the serving layer independently. However, we’ve found in practice that our fanout is not extreme enough to merit separating the serving layer, especially given the bandwidth available on modern hardware.

[nick0garvey]

Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot.

I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

[EdwardDiego]

Okay, I can see that point, but is it worth the additional latency between broker and Bookie?

> I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.

[rad_gruchalski]

> Okay, I can see that point, but is it worth the additional latency between broker and Bookie?

It might depend on what you're ingesting and how much. Being able to independently scale ingest and storage is a good alternative to have. It's not only ingest though. It's also consumption. As it stands, having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory. In that sense, reprocessing historical data is pretty difficult. There's a lot of complexity hidden in additional Druid, HDFS installations or shoe-horned object storage with their own indexing to support access to historical data semi-fast.

[EdwardDiego]

> having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory.

I don't know too much about Kafka's internals, but that's my not experience of reading several terabytes of data from a Kafka topic. Memory didn't blow out, although we did burn through IOPs credits.

[rad_gruchalski]

Good remark. When there's one consumer or multiple consumers hanging on roughly to the same offset (using the same memory mapped segment). Having many consumers hanging on widely different offsets will cause many segments sitting in RAM.

Edit: Kafka apparently does not store a complete log segment in memory, only parts but having many consumers may lead to a lot of churn or a lot of memory consumed. Maybe this is getting better.

[skyde]

Latency is actually reduced compared to kafka because when you have 3 replica you just need to wait for a response of the fastest of the 3 replica instead of waiting for a response of the slowest (master) of the 3 replica.

[manigandham]

Confluent is already working on this: https://www.confluent.io/project-metamorphosis/infinite

The benefit is not just independent scaling of resources, but more useful features like archiving and reading from object storage with infinite history, and faster and more reliable data replication across regions.