[opportune]

Not only have I worked on some of the highest throughput workloads in FAANG, the ones I’ve worked on involve consuming from async queues (including Kafka) as a first class feature and use additional queueing under the hood to make the system work.

Kafka is overkill for the vast majority of users, including many in FAANG, and at companies in FAANG that I’ve been at, usually either an internal async queue or a bespoke solution for the problem is used instead. But more importantly, async queueing via Kafka-like technology itself is way over-applied. Polling a single node RDBMs with a horizontally scaling cache in front of it is a fine way to propagate eg config changes at massive scale. And many of the “real-time ingestion” use cases that people love using Kafka for are much better off simply being synchronous, with engineering effort instead focused on the ability to rapidly scale consumers up and down.