[jauntywundrkind]

I'd also strongly recommend this view of how Kubernetes uses cgroups, showing similar drill downs for how everything gets managed. Lovely view of what's really happening! https://martinheinz.dev/blog/91

I've been a bit apoplectic in the past that cgroups seemed not super helpful in Kubernetes, but this really showed me how the different Kubernetes QoS levels are driven by similar juggling of different cgroups.

I'm not sure if this makes use of cpu.max.burst or not. There's a fun article that monkeys with these cgroups directly, which is neat to see. It also links to an ask that Kubernetes get support for the new (5.14) CFS Burst system. Which is a whole nother fun rabbit hole of fair share bursting to go down! https://medium.com/@christian.cadieux/kubernetes-throttling-... https://github.com/kubernetes/kubernetes/issues/104516

[msarnowicz]

Reading through the description of how cgroups are used in Kubernetes, I can see some similarities and some differences as well. It is interesting to compare the approaches.

We chose not to use cpu.weight, and instead divide the host explicitly using cgroups (slice in systemd). We put Standard VMs in dedicated slices to keep them isolated and let several Burstable VMs share a slice. This provides a trade off between the price of the VM and resource guarantees.

We use cpu.max.burst to allow the VMs to "expand" a bit, while we understand that this creates a "noisy neighbor" problem. At the same time there is a minimum guarantee of the CPU. The cgroups allow for all those knobs and give a lot of control. Combining them in various ways is an interesting puzzle.

[__turbobrew__]

cpu.max.burst increases the chances of noisy neighbours stealing CPU from other tenants.

I run multi-tenant k8s clusters with hundreds of tenants and it fundamentally is a hard problem to balance workload performance with efficiency. Sharing resources increases efficiency but in most cases increases tail latencies.

[jeffbee]

If you use k8s qos levels "guaranteed" cpu resources will be distinct — via cpu sets — from the ones used by the riff-raff. This is a good way to segregate latency-sensitive apps where you care about latency from throughtput-oriented stuff where you don't.

[__turbobrew__]

Guaranteed QoS isn’t perfect:

1. Neighbours can be noisy to the other hyperthread on the same CPU. For example, heavy usage of avx-512 and other vectorized instructions can affect a tenant running on the same core but different hyperthread. You can disable hyperthreading, but now you are making the same tradeoff where you are sacrificing efficiency for low tail latencies.

2. There are certain locks in the kernel which can be exhausted by certain behaviour of a single tenant. For example, on kernel 5.15 there was one global kernel lock for cgroup resource accounting. If you have a tenant which is constantly hitting cgroup limits it increases lock contention in the kernel which slows down other tenants on the system which also use the same locks. This particular issue with cgroups accounting has been improved in later kernels.

3. If your latency sensitive service runs on the same cores which service IRQs, the tail latency can greatly increase when there are heavy IRQ load, for example high speed NIC IRQs. You can isolate those CPUs from the pool of CPUs offered to pods, but now you are dedicating 4-8 CPUs to just process interrupts. Ideally you could run the non-guaranteed pods on the CPUs which service IRQs, but that is not supported by kubernetes.

4. During full node memory pressure, the kernel does not respect memory.min and will reclaim pages of guaranteed QoS workloads.

5. The current implementation of memory QoS does not adjust memory.max of the burstable pod slice, so bursable pods can take up the entire free memory of the kubepods slice which starves new memory allocations from guaranteed pods.

Dont even get me started on NUMA issues.

[jeffbee]

There isn't any way on Linux to deal with processes that create dirty pages. It is folly to try. The only way to deal is to put I/O stuff on a whole box/node by itself, and outlaw block I/O on all other nodes.

[immibis]

Can't find the article where I first read it (something like "Queuing theory for software engineers") but average latency increases as, IIRC, serving time ÷ (1 - utilization). Get half as close to 100% utilization, and you double your average latency. A system with 87.5% utilization has double the latency as at 75%. At 100% it's infinity (averaged over infinite time - on shorter timescales it's an unpredictable scale-free random walk).

This is fundamental - the closer utilization is to 100%, the higher the chance a newly arriving work item has to wait for one that's already running, and several already in the queue. What's astonishing is how steep that curve is. At 95% utilization the average queue length is 20 tasks. At 99% it's 100 tasks. At 99.9% it's 1000 asks. If you find yourself at 98% utilization, you should not think "nice - in fully utilizing the server I paid for" - you should buy another server and lower it to 49%. (Or optimize the code more)

One way to deal with this is to have separate low-latency and high-latency queues. You can then run low latency tasks at say 50% utilization and fill up idle time with high latency tasks. Presuming and you actually want the HL tasks to ever get done, you can't guarantee 100% utilization, but you can get arbitrarily close as long as there's high-latency work to do. I have no idea whether this is something Kubernetes can do. You can of course have more than two priority levels.

This applies everywhere there's a queue, which is basically everywhere there's s contended resource. Hyperscalers know this. It's even been theorized that S3 Glacier is just the super low priority disk access queue on regular AWS servers (but Amazon won't tell us).

[remram]

Maybe one of these? https://dzone.com/articles/queuing-theory-for-software-engin... https://medium.com/@quebostina/stack-and-queue-are-two-of-th...

[hinkley]

I suspect you can only really count on neighbors to take care of their own. Anything else they see will be taken as an entitlement.

So for instance if you run three processes for the same customer, can you set them to use the same cpu slices and deal with one of their apps occasionally needing a burst of CPU?

[__turbobrew__]

Sure in theory you could do that, but kubernetes does not support overriding the top level cgroup a pod is assigned to.

[msarnowicz]

Thank you, that is a good perspective, too!