Hacker News new | ask | show | jobs
by TheDong 3192 days ago
Depends on several other variables.

If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured.

If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few

The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container.

"containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most.

The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k.

Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total.

Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu

This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.

2 comments

What happens when you hit the limit? Thrashing swap or oom kills? (And what kind of overcommit policy did you use?)
I didn't hit any limit, but since I don't have swap it would likely be oom kills.

The first part of my post is speculating, the second part I ran an arbitrary number and observed resource usage to allow extrapolation, but didn't hit a limit.

Thanks for reproducing the experiment. :) I sent a message to tianon asking him if he remembers what the original numbers were. He told me long ago, I think it was around 1000. This was well before the go shim or dockerd existed.
thanks, this was interesting! That's better than I expected.