[cstrahan]

"Synchronized demand is the moment a large cohort of clients acts almost together."

Imagine everyone in a particular timezone browsing Amazon as they sit down for their 9 to 5; or an outage occurring, and a number of automated systems (re)trying requests just as the service comes back up. These clients are all "acting almost together".

"In a service with capacity mu requests per second and background load lambda_0, the usable headroom is H = mu - lambda_0 > 0"

Subtract the typical, baseline load (lambda_0) from the max capacity (mu), and that gives you how much headroom (H) you have.

The signal processing definition of headroom: the "space" between the normal operating level of a signal and the point at which the system can no longer handle it without distortion or clipping.

So headroom here can be thought of "wiggle room", if that is a more intuitive term to you.

[fuckaj]

Is the pragmatic solution to return 503 and have clients back off.

Or, if possible make latency a feature (embrace the queue!). For service to service internal stuff e.g. something like a request to hard delete something, this can always be a queue.

And obviously you can scale up as the queue backs up.

I do love the maths tho!