[skyde]

I think you are completely missing the point. Those kind of system need to be designed in layer. You can host all 3 layer on each of your VM instance if you want but they should not be mixed in the same process.

One layer BookKeeper provide an abstraction similar to HDFS. That is it provide file that are horizontally scalable in size and throughput and reliable append only files.

Pulsar is a service built on top of BookKeeper but could run on top of HDFS or something like Amazon S3 ...

And is only responsible for making sure there is only one writer per BookKeeper file even if multiple process try sending request to Pulsar to write to the same partition. It also try to balance request across all the brokers.