Hacker News new | ask | show | jobs
by NegatioN 1727 days ago
This might be more fit for StackOverflow, but I have a related question.

I have a Go application that runs in Kubernetes, where memory usage steadily increases until it's at around 90% of the cgroup limit, where it seems to stabilize. As far as I can tell, Go GC uses the container memory limits to navigate it's total memory usage (this might be the fault of the OS not reclaiming what Go has already freed(?)).

However, my issue is that in this app, I also call out to cGo, and do manual memory allocations in C++ every 10-30minutes. This works well, except when the container is stabilized at a high memory usage, and my manual allocation brings it over the limit, thus forcing kubernetes to terminate it. (These allocations should as far as I know not be leaking. For a short while, I have two large objects allocated, and 99.9% of the time it's only one)

So, what I'd ideally want is to be able to specify a target heap size for GoGC, and then have a known overhead for the manual allocation. But as far as I'm aware, this isn't possible (?)

Does anyone have any experience with something like this, or see any obvious avenues to pursue to solve the termination issue?

1 comments

Since Go seems to respect the memory limit, you could try using syscall.Setrlimit to set an artificially lower limit that you know will leave enough room for your other allocations. Have you tried playing with the GOGC environment variable from the runtime package? Maybe you could also manually collect a memory profile with runtime.MemProfile and call runtime.GC() if needed, but I've never done anything like this, just throwing out ideas I would probably try