Hacker News new | ask | show | jobs
by j_baker 5387 days ago
If your queues are getting that backed up, you're either facing bigger problems than your queueing system (most likely workers being down), or you're big enough to afford more machines and/or a custom solution (such as kestrel).
1 comments

The parent post referred to using Redis as a multi-purpose store. If there is a bunch of other non-queue data in Redis and the setup is only expecting it to use 1GB or so, there's likely not a giant amount of room for queue entries left. While everyone is sleeping, some crashed workers combined with broken or poorly configured monitoring can fill up queues very quickly. Been there, done that. Either the OOM hits and there is some data loss or the swap hits and brings everything down.