[ible]

I'm not clear on why you would use redis to get transactions. Wouldn't sending a single message to kafka that represents the whole transaction with acks=all have the same effect? Then instead of pulling from redis like it is a queue, you can just do that from kafka.

[ariskk]

This is exactly what we did initially. Really early on (before any rate limiting was in place), a few spam accounts followed 100K people, created lots spam content etc. Encapsulating those deletions started yielding messages bigger than the default max kafka message size (1MB). Additionally, this method had a few side effects on the downstream processors. We could of course increase the limit, but we decided to deal with the problem at its core.