[dyanacek]

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

[tome]

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

[8note]

A queue distributes the latency increase to all requests whereas a stack only increases the latency for some requests when you're overloaded.

This means that if you catch up to the incoming new requests, a queue keeps every request running slower due to the time spent in the queue.

The steady state stack on the other hand, gives the same couple items worse and worse latency, while all the new items go back to normal.

Chances are the long latency requests will be retried, and there's a roughly fixed number of items to be retried, so you don't have to worry too much about the ones stuck in the stack. For a queue, the retries lengthen the queue, and that waiting time is added to every item, making them more likely to retry too, lengthening the queue further etc

Having the stack leak items at the bottom gets the same benefit - you don't have to fulfil them, but if you can get the items back out of the stack quickly, it's still worth working on and completing them before the client needs to send a retry. The more of them you get in, the more 9s you get to look more like your p50 than your p100

[lokar]

Consider the two steady states:

You are keeping up and the queue is mostly empty, the order does not matter

You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.

[tome]

Oh, I think I see. With a queue the failure state is that everything times out. With a stack you risk sacrificing only the oldest requests and keep the newest alive.

[lokar]

And you end up not servicing some random sample that get buried in the stack before you can get to them.

[oriolid]

It guarantees that in overload situation the requests that get handled are handled quickly. In the same situation a FIFO would grow until all requests are really slow or time out without increasing throughput. The reason to keep old requests in LIFO instead of dropping then right away is that they can be served when load drops, just in case there's still someone waiting for the page to load.

[tyingq]

I suppose the older requests are less likely to have actual people doing a page reload/retry that stacks up yet more demand.