[jlgreco]

How would you design a system that didn't have this problem at the individual node level?

[philjr]

A similar design to TCP/IP - sequence numbers, acknowledgements & resending capability on sender side

[dclusin]

Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.

[jlgreco]

So the sender just keeps trying until it gets an ACK. Fair enough, though I imagine you'd want to run multiple nodes even with this.

[dclusin]

Correct. Reliability and Availability are two separate but related goals.

[onetwothreefour]

This is a solved problem in most messaging protocols.