[Birch-san]

You may be experiencing the same bug as we did: memory freed by the GC of a containerised JVM was not able to be returned to the host machine.

IIRC it was due to a bug in glibc >= 2.1. Something about how mallocs are pooled. IIRC you need to tune it to be <= num of physical threads. Usually people advise 4 or 2.

  # openjdk 8 bug: HotSpot leaking memory in long-running requests
  # workaround:
  # - Disable HotSpot completely with -Xint 
  # - slows the memory growth, but does not halt it:
  MALLOC_ARENA_MAX=4

So, ensure that your java process is launched with that environment var (so, export it in the same shell, or precede your java command with it).

If you happen to be using Tomcat, I recommend putting:

    export MALLOC_ARENA_MAX=4

into:

    /usr/local/tomcat/bin/setenv.sh

As for how much memory you allocate to your containers: as of JRE 8u131 you can make this far more container-friendly:

    -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

This is equivalent to saying:

    -XX:MaxRAM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)

https://github.com/moby/moby/issues/15020 https://github.com/docker-library/openjdk/issues/57 https://bugs.openjdk.java.net/browse/JDK-8170888