[neovintage]

I believe Redis has a great developer experience. It’s easy to get set up and use. When I see the work on Redis streams, I think its going bring a much better getting started experience for developers that want to start using evented architectures. This might be a turning point where we see more developers and applications utilizing those types of architectures. In the end, this might give Apache Kafka a run for its money, or not, who knows. I've tried using Apache Kafka and it can be a bear to set up given its dependencies and to administer.

Put another way, I think Kafka solved a problem for enterprises and was a tops down approach to the problem. Redis streams is a bottoms up approach to implementing evented architectures. Maybe there's room for both products in the market?

[antirez]

Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice.

For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation they have on the site about the design, to get the higher level picture and combine this with my own ideas about fixing the fact Redis was lacking a "log" data structure.

Pub/Sub + other data structures were not able to provide time series and streaming, but yet Redis streams remain an ADT (Abstract Data Structure), while Kafka is a tool to solve a very specific business case. So the applications have some intersection, but are also very very apart.

For instance you can create infinite small keys having streams, so Redis is good for many IoT usages where you receive data from many small devices. Redis Streams also stress on range queries, so that you can combine the time series with millisecond-range queries.

However, yes, the fact that I added also consumer groups is a way to put this "80% streaming" into a more usable streaming systems more similar to Kafka, for the use cases where:

1) The memory limits.

2) The speed.

3) The consistency guarantees of Redis make sense.

However at the same time, it was a great challenge and pleasure to do what I always try to do, that is to create an API for developers thinking like I'm designing an iPhone, and not some terrible engineering thing which does what it should but is terrible to use (I'm not referring to Kafka that I do not know). So I really hope that what you say "easy to setup and use" will be what developers will feel :-)

[j_s]

I'm a big fan and I hope you see this comment before the edit window passes (unlikely, unfortunately):

You need to get yourself some whitespace in this reply!

[antirez]

Thanks! Done :-)

[jadbox]

What level of guarantees will Redis Streams have during power failure? Can I configure if events are close to immediately persistent to disk VS served from memory and occasionally paged to disk?

[manigandham]

Redis streams are just another data type and follow the same persistence settings that have been available to Redis: https://redis.io/topics/persistence

You can use either RDB: snapshots of the entire dataset at some interval and/or AOF: logging of all changes fsync on every write, every second or leaving it to the OS.

[misterbowfinger]

> that is to create an API for developers thinking like I'm designing an iPhone

Albeit my request is quite selfish, I really would love to hear/read more about your thoughts on designing APIs. My experience using Redis has been excellent and I'd to be able to replicate that sense of design in the systems I build as well.