[nachexnachex]

Properly scaling queue consumers is a problem I've spent a lot of time on in the last few years. Working on a messaging platform with highly variable traffic, including close to zero during the night, means that capacity provisioning according to the max will be very costly, and lead to a lot of frustration when you are saturated anyway.

Indeed you need backpressure but the traditional methods (CPU usage or similar metrics) are difficult because many consumers aren't high on those metrics --imagine a messaging plaform, pure IO. Also you'd have to tailor to the consumer itself and that's difficult, which is what you mention on the next-to-last paragraph.

In the end I helped solving it by scaling based on queue size and input/output rates, agnostic to the consumer itself, but with the hypothesis that you can scale consumers linearly (or at least monotonically, some sublinearity is allowed). The queue scaler watches for incoming and outgoing traffic on the queue, plus items on the queue itself, and it can scale from 0 to 11 in seconds for gusts, then shutting everything down.

It's a satisfying problem to work on, but its proper solution demanded quite the investigation. Now every queue we've got in the system is managed by this autoscaler -- except when we can't ensure linearity.

[groundzeros2015]

> capacity provisioning according to the max will be very costly

I’m skeptical. You can support a pretty massive messaging system with one box.

What did you try? What went wrong?

[nachexnachex]

Similar to what gp mentioned, max latency. You can tune your consumers to burn through the queue and push messages, but when you have latency constraints e.g. a very tight deadline the first few messages in the queue might achieve it but not the 20000th.

When you want to switch from no processing to say 2M messages than need to be sent within 5 minutes it's difficult to appropriately provision without fast scale out.