Hacker News new | ask | show | jobs
by divideby0 3108 days ago
I'm not quite certain I understand that last part. To the best of my understanding, Kafka's design is such that consumer ingestion is completely decoupled from production throughput. All messages from the producer are first copied to disk and then zero-copied from disk to network to the consumers at the byte array level. If a consumer falls behind, it has its own independent offset stored on the broker that keeps track of where in the byte array (or log) it left off. This, by design, allows Kafka to handle different profiles of consumers and even have consumers drop off entirely and later join to catch up. But perhaps I'm missing something about what you're saying.