[nosefrog]

Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.

[esafak]

Care to share more about the issues?

[nostrademons]

Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are:

1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be as simple as "it was the 20th request to arrive during a period of high demand, and backends couldn't keep up". The whole point of queues is so you can gracefully handle this case without overprovisioning your backends by a factor of 20x, but if the user is still going to consider this a miss anyway, you have to overprovision the backends anyway. There isn't really another way to handle this other than having spare backend capacity. Also note that in many cases the user hitting "refresh" doesn't cancel the existing queued job, it just adds another one to the queue. Which brings us to...

2.) They can turn simple failures into cascading failures. There were several postmortems that went something like "Service X became overloaded because of an unexpected flood of requests, leading to several individual replicas shutting down. This led to more requests being routed to the remaining replicas, which overloaded them too and led to all of Service X going down. When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure. SRE had to limit requests upstream and manually drain all job queues and bring Service X back cluster by cluster to restore service health."

The root principle here is that any distributed system needs a concept of backpressure. When critical downstream dependencies are overloaded, they need to pass this information back up the stack to the entry point, which needs to start denying requests from the user or do a simpler fallback that doesn't put load on the overloaded service. Naive queueing does not work, because the requests are still sitting there in the queue waiting to overload the downstream service once it becomes available again.

You can bolt backpressure onto a job queue system (by eg. rate-limiting requests to a service that has just come back up, or rate-limiting based on response time, and/or falling back to simpler algorithms), but at that point, it's a backpressure system, not a job queue. The semantics are very different from a system that guarantees eventual delivery, just not sure when. You need to be able to handle partial failures and adapt with different algorithms at multiple points within the system.

[mike_hearn]

I also worked on services at Google, both as an SRE and a SWE.

I think one subtlety worth bearing in mind here is that Google, at least during my tenure (2006-2014) didn't actually have a proper message queueing system outside of Gmail, which was used only for email delivery. It offered engineers:

1. RPC with infinite backoff/retry (a fun default).

2. Batch jobs that processed files.

but there was no equivalent to a standard enterprise MQ product like ActiveMQ, Artemis, Oracle AQ and so on.

So when we talk about "job queues" it's worth being very precise about what is meant. A standard enterprise message queue broker doesn't have problems with overload because workers pop work off the queue at whatever speed they can operate. The problem of cascading failures and services coming back only to be immediately overloaded again was a problem caused by the design of Google's infrastructure, in which RPCs would back up in memory in the clients and be retried in a loop until the service came back. So of course this required a lot of custom work to create proper backpressuring, which wasn't normally done and especially not on interactive serving paths, leading to this kind of repetitive failure mode. It was all made harder by the practice of making everything fully async, so thread pool sizes also didn't exert backpressure.

Looking back on my time there, one of the pieces of "normal" enterprise infrastructure that I really think Google could have benefited from was a proper JMS compliant scalable message broker. You can buy these - the Oracle Database has one - but Google neither bought one nor developed its own.

Probably they have long since rectified this oversight.

[cyberpunk]

Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each.

It’s a hard/interesting problem, and harder still once you’re running across a lot of machines — but if they all get slower under the load it turns out that it’s easier to scale / work out a good balance of idle capacity to guarantee x time sla under x requests.

Fast ramp up for additional capacity is important too, but less so if you know to start the process once median execution time drops to some % of your worst target

[eru]

> Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each.

Well, memory is one of these resources that you are often locked on.

If you have enough memory, running 20x the load just goes 20x slower. But if memory is congested, then this can go arbitrarily slower than running your jobs one after another. Eg when you are swapping to a spinning disk.

[cyberpunk]

I’ve never enabled swap on a production system in ~25 years, so i have to say it’s not really my experience.

Shared memory can be a problem, sure, which is why I don’t generally use that either. Erlang, of course, generally does not use shared memory outside of some cases with ETS etc which must be used carefully but i’d rather solve those problems myself.

Concurrency systems in other languages i’ve written, for example Go, there are ways to architect to avoid it also. I’d rather go slightly slower and copy be value than have to solve mutex contention and trying to make everything atomic and so on. YMMV, I don’t work in HPC just large complex busy systems.

[eru]

To make my point more abstractly:

Multiple tasks can share the CPU and just get a bit slower. But if you are out of RAM, you are better off running one thing after another.

Whether you hit swap or OOM was a distraction.

[nachexnachex]

Properly scaling queue consumers is a problem I've spent a lot of time on in the last few years. Working on a messaging platform with highly variable traffic, including close to zero during the night, means that capacity provisioning according to the max will be very costly, and lead to a lot of frustration when you are saturated anyway.

Indeed you need backpressure but the traditional methods (CPU usage or similar metrics) are difficult because many consumers aren't high on those metrics --imagine a messaging plaform, pure IO. Also you'd have to tailor to the consumer itself and that's difficult, which is what you mention on the next-to-last paragraph.

In the end I helped solving it by scaling based on queue size and input/output rates, agnostic to the consumer itself, but with the hypothesis that you can scale consumers linearly (or at least monotonically, some sublinearity is allowed). The queue scaler watches for incoming and outgoing traffic on the queue, plus items on the queue itself, and it can scale from 0 to 11 in seconds for gusts, then shutting everything down.

It's a satisfying problem to work on, but its proper solution demanded quite the investigation. Now every queue we've got in the system is managed by this autoscaler -- except when we can't ensure linearity.

[groundzeros2015]

> capacity provisioning according to the max will be very costly

I’m skeptical. You can support a pretty massive messaging system with one box.

What did you try? What went wrong?

[nachexnachex]

Similar to what gp mentioned, max latency. You can tune your consumers to burn through the queue and push messages, but when you have latency constraints e.g. a very tight deadline the first few messages in the queue might achieve it but not the 20000th.

When you want to switch from no processing to say 2M messages than need to be sent within 5 minutes it's difficult to appropriately provision without fast scale out.

[tzs]

> Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow.

This reminds me of some research I read about in the 1980s or 1990s on perceptions of the speed of command line commands. If command time varied over a range from nearly instantaneous to say 100 ms fairly uniformly through that range, people would perceive the system as overall being faster when the researchers added a variable delay to all the commands that made them all take 100 ms.

Humans apparently really like consistency.

> When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure

...and this reminds me of something else, from around 1983. I was working at a small Unix workstation maker. The guy in the office across the hall found one morning that the battery for the clock on his workstation had died, and the system time had come up after boot as the Unix epoch.

He shut down, put in a new battery, booted, and then set the clock to the current time, 13 or 14 years after the epoch.

Almost immediately his hard disk light came steadily along, and he could hear the disk furiously seeking, and the system became completely unresponsive.

It turned out AT&T cron in the early '80s wasn't smart about time changes. It had tried to all at once every cron job that should have run in the 13 or 14 years that the time just jumped.

[lkjdsklf]

The solution to the complexity of any queuing system is to add another queue