I'm looking at Redis Streams and it seems to lack the partitioning component of Kafka. One of the properties that we rely on is the ability to manually assign partitions such that a worker assigned partition 0 of one topic, will also be assigned partition 0 of other topics that it consumes from.
Simplicity is of course a goal, but this may mean we have to sacrifice some features when Redis Streams is used as a backend.
I'm glad you like Celery, this project in many ways realize what I wanted it to be.
In Faust we've had the ability to take a very different approach towards quality control. We have a cloud integration testing setup, that actually runs a series of Faust apps in production. We do chaos testing: randomly terminating workers, randomly blocking network to Kafka, and much more. All the while monitoring the health of the apps, and the consistency of the results that they produce.
If you have a Faust app that depends on a particular feature we strongly suggest you submit it as an integration test for us to run.
Hopefully some day Celery will be able to take the same approach, but running cloud servers cost money that the project does not have.
Simplicity is of course a goal, but this may mean we have to sacrifice some features when Redis Streams is used as a backend.
I'm glad you like Celery, this project in many ways realize what I wanted it to be.