[manigandham]

Great to see. We actually ended up using Redis for our event stream after trying Kafka and the rest. Using the list extension module multiple list pop functions lets us get to 2gbps of throughput on a single redis node with AOF persistence. Using streams would make things even simpler and faster.

[somethingsimple]

Did you have issues with Kafka? Curious to hear because I’m about to try and start using it for something at work.

[manigandham]

It's just much more work to install and maintain Kafka, and it has issues with load balancing and recovery due to the design tying cluster ownership to partition data. With AOF persistence and a replica, Redis is durable enough for us and extremely fast with no maintenance.

If you absolutely need Kafka then it's still a good option, although I'd recommend looking at Apache Pulsar [1] for a better design. It separates storage and compute for better performance and scalability while giving you features like per-message acknowledgements.

1. https://pulsar.apache.org