|
|
|
|
|
|
by immibis
405 days ago
|
|
|
Can't find the article where I first read it (something like "Queuing theory for software engineers") but average latency increases as, IIRC, serving time รท (1 - utilization). Get half as close to 100% utilization, and you double your average latency. A system with 87.5% utilization has double the latency as at 75%. At 100% it's infinity (averaged over infinite time - on shorter timescales it's an unpredictable scale-free random walk). This is fundamental - the closer utilization is to 100%, the higher the chance a newly arriving work item has to wait for one that's already running, and several already in the queue. What's astonishing is how steep that curve is. At 95% utilization the average queue length is 20 tasks. At 99% it's 100 tasks. At 99.9% it's 1000 asks. If you find yourself at 98% utilization, you should not think "nice - in fully utilizing the server I paid for" - you should buy another server and lower it to 49%. (Or optimize the code more) One way to deal with this is to have separate low-latency and high-latency queues. You can then run low latency tasks at say 50% utilization and fill up idle time with high latency tasks. Presuming and you actually want the HL tasks to ever get done, you can't guarantee 100% utilization, but you can get arbitrarily close as long as there's high-latency work to do. I have no idea whether this is something Kubernetes can do. You can of course have more than two priority levels. This applies everywhere there's a queue, which is basically everywhere there's s contended resource. Hyperscalers know this. It's even been theorized that S3 Glacier is just the super low priority disk access queue on regular AWS servers (but Amazon won't tell us). |
|
|