[tootie]

Isn't the ideal solution to make the throttled system faster? Like autoscale horizontally and/or vertically, sharding or just writing better code? Everything in this article is about to cope with back pressure but solving is frequently possible.

[jcgrillo]

I'll bet your backpressure mechanism can react at least an order of magnitude faster than your scaling mechanism.

[vrosas]

Serverless systems are pretty decent at scaling quickly these days. The problem is rarely lack of servers in my experience, though. You usually run out of database connections or some other bottleneck first.

[immibis]

In other words, you run out of database servers, or your connection limit is too low so you don't fully utilize your database servers.

[coldtea]

>Like autoscale horizontally and/or vertically, sharding or just writing better code?

Sometimes you just have a server, not some elastic horizontal setup, or a budget for vertical scaling.

And "writing better code" is an optimization thing, not a general way to handle back-pressure (you can have back-pressure in a fully optimized program too, and even for non-optimal code you don't want to just have to iterate and refactor/improve the code every time there's back-pressure).

[toast0]

In some ways, yes, a solution to your system is throughput constrained is to remove the constraint.

But, there's a cost to that, it may be development time, capital, or operational expense. Certainly, sometimes you spend a few minutes replacing a bad sort with a much better sort and get immediate and large benefits. But often it's the case where rare conditions cause loads that are too expensive to handle immediately.

Having backpressure setup and monitored in advance also comes at a cost, but allows you to make specific choices about how the system works in overload, and allows you to monitor the overload conditions, and hopefully/maybe gives you some feedback about your overall capacity.

For example, many systems have worse throughput when overloaded. Having a strict concurrency limit prevents throughput loss from context switching, and the errors (or simply effects from queueing) can propigate as backpressure. Monitoring active threads gives a sense of available capacity, monitoring time spent with threads at max or depth of queue / number of requests rejected gives a sense of unmet demand.

[photonthug]

In practice, yeah, this is the fix in most systems because in a microservice context it would feel gross to have a slow consumer reach out and throttle a fast producer.

It’s still important to understand though because autoscaling has its own back pressure that you run into eventually. Accounts may have quotas or regions might be out of a given instance type, or out of a type at your preference spot price.

[oivey]

It is in general never possible because if your load can scale infinitely, then there won’t ever be enough compute to satisfy it. In practice, you may be able to apply some assumptions to bound the amount of compute necessary. It is still almost always good to have a plan on how to shed load if your assumptions fail. Systems built without this are prone to failing catastrophically.

[convolvatron]

it doesn't matter. as long as there is a rate mismatch over a sufficiently long timescale, queues will build up and consume memory, latency will increase, and depending on how things are structured overall throughput will go down because of scheduling overhead and memory pressure.

this is actually an inherent problem _within_ horizontally scaled services that have cross dependencies.

so no, you should really implement backpressure.

[tootie]

"sufficiently long timescale" is long enough to figure out a solution :)

But seriously the question was "ideal solution". Not everyone can do it but just right-sizing compute and removing or offloading bottlenecks is entirely possible. So is just throwing in the towel and discarding excess load. Reddit does that all the time and their IPO was a big success. You can also scale to accommodate massive headroom by just throwing money at it. There are plenty of systems where response times are mandatory and any back pressure mitigation that slows traffic is unacceptable.

[_3u10]

Ideally yes, in practice… no.

In reality you hit amdahl’s law pretty quickly.