[Spivak]

Hard disagree, Kafka is one of the simplest lowest maintenance tools for this with excellent language support and would probably be the first choice for anyone not paying $cloud_vendor for a managed durable queue.

The first step in building a reliable logging system is setting up a high write throughout highly available FIFOish durable storage. Once you have that everything else gets a lot easier.

* Once the log is committed to the durable queue that's it the application can move on secure the log isn't going to get lost.

* Multiple consumer groups can process the logs for different purposes, the usuals are one group for persisting the logs to a searchable index and one group for real time altering.

* Everything downstream from Kafka can be far less reliable because it's just a queue backup.

* You can fake more throughout then you actually have in your downstream processors because it just manifests as a lagging offset.

[rewmie]

> Hard disagree, Kafka is one of the simplest lowest maintenance tools for this (..)

You sound like you've been using an entirely different project named Kafka, because the Kafka everyone uses is renowned among message brokers for its complexity and operational overhead.

[Spivak]

I might be, it's one of the lowest touch services we run. But we aren't doing the "Kafka all the things" model where every single little app is hooked into it for generic message passing but simply logs go in, logs go out, nothing else.

The business logic message passing goes through Rabbit because we wanted out of order processing, priority routing, retry queues, blah blah.