[fpgaminer]

> Were the system pods using all that memory or just reserving it?

700MB was the sum of all the requested minimum RAM for all those service pods. So yeah, you're probably right that they're ceilings of sorts. Still it's a bit crazy to see a logging service, who's job is merely to haul logs off to a different server, requesting 200MB.

I'm also bewildered by Container Optimized OS's memory consumption. IIRC it was 500MB+ bare; doing nothing. As reported by top. I forget which, but I stood up either Debian Stretch or Ubuntu 18.04 and it was only ~200MB with Docker installed.

[puzzle]

The logging service numbers are easily explained: the remote server might have transient failures, so the forwarder will cache stuff in memory (the alternative is to just stop reading the logs, but then you risk losing messages if the pod dies in the meantime). You complained about Go, but it doesn't help that the fluentd agent is written in Ruby. There's a new Go rewrite of it, but I don't think GKE or others use it yet.

Was COS a standalone GCE instance or GKE? In the latter case, memory will be used by the usual suspects: fluentd, kubelet, kube-proxy, docker, node-problem-detector. For both, there are also a few Google daemons in Python (ugh).