Hacker News new | ask | show | jobs
by pvlak 2447 days ago
Will the WriteService(Aggregator) make sense for environments having thousands of machines(not in millions) and they are all within the DataCenter. In our company, we are moving away from this design of having aggregators, to directly writing to Storage whereever possible, as it reduces the message loss.

On the volume of metadata held by Producers, will there be any significant difference between holding WriteService & LogDevice meta.

1 comments

The devil is in the details probably, but if you have a single datacenter and all writes are coming from thousands of machines ("edge"), yes, it may make more sense to set up a single LogDevice / Kafka cluster and have all the edge hosts write to it directly.