[nitrogen]

Kept indefinitely for future re-consumption.

Are there any stories of this saving someone from a potential disaster? My experience has been that this only causes bugs, such as resending hundreds of thoudands of out-of-date emails.

[dcminter]

It's not usually intended as a disaster recovery strategy; it's more often intended so that the event stream can be ingested in future by completely new consumers.

Retention policies and compaction exist where you don't actually want to keep the data, but the capability is one of the distinguishing features.

You can make an event log look like a database or a queue or a cache - but if you do that you should definitely consider whether you are using the right tool for the job.

[morelisp]

Independently of application logic (which also sometimes uses it), it's the primary mechanism in Kafka for handling high-throughput consumers while still recovering from failure. Consumers grab a batch of e.g. 1000 events, checkpoint every X seconds while processing them; if they die the events are still there and they restart from the last checkpoint.

It also means every message in Kafka is "addressable" via topic/partition/offset which lets you refer to "foreign" messages etc.

[jpgvm]

Being able to replay the contents of a Kafka log is a very common recovery mechanism for many distributed systems and is used all the time.