https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
I greatly prefer redis streams. Not all the same features, but if you just need basic streams, redis has the dead simple implementation I always wanted.
Not to mention you then also have a KV store. Most problems can be solved with redis + Postgres
Don't, under any circumstances, let it come into contact with an untrusted network. Anyone who can connect to your Redis service gets arbitrary remote code execution.
Honestly no, which was my favorite part. So easy to spin up, "just works" the way you expect. It's night and day from a lot of the other solutions in the ecosystem.
1. Core NATS, an ephemeral message broker. It's a lightweight pub/sub system where routing of messages is based on wildcard paths. All in memory, lightning fast, extremely lightweight. You can use it for RPC, queues, broadcasting, file transfer, anything.
2. JetStream, which is a Kafka/Pulsar-like log built on top of Core NATS. Streams are indexed, meaning there's no strong need for partitioning or consumer groups, since readers can efficiently filter the stream by interest (you can still partition for write performance). Supports both durable and ephemeral consumers, in-memory streams, ack/nack, priority groups, deduplication, exactly-once delivery, hierarchical clusters, offline clusters ("leaf clusters"), mirroring, etc.
I often find it difficult to explain the magic of NATS. It's a communication model that doesn't really exist anywhere else, as far as I've seen. The closest might be ZeroMQ.
JetStream could be explained as "Kafka for people who don't want to administer Kafka". It's very low-maintenance, very easy to use, feels super lightweight, and still offers much of the performance and reliability of Kafka, as well as a much richer feature set that maps better to what people may want from a streaming log.
I came here to say just that. Nats solves a lot of those challenges, like different ways to query and preserve messages, hierarchical data, decent authn/authz options for multi-tenancy, much lighter and easier to set up, etc. It has more of a messaging and k/v store feel than the log Kafka is, so while there's some overlap, I don't think they fit the exact same use cases. Nats is fast, but I haven't seen any benchmarks for specifically the bulk write-once append log situation Kafka is usually used for.
Still, if a hypothetical new Kafka would incorporate some of Nats' features, that would be a good thing.
https://news.ycombinator.com/item?id=43783452