If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured.
If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few
The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container.
"containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most.
The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k.
Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total.
Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu
This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.
I didn't hit any limit, but since I don't have swap it would likely be oom kills.
The first part of my post is speculating, the second part I ran an arbitrary number and observed resource usage to allow extrapolation, but didn't hit a limit.
Thanks for reproducing the experiment. :) I sent a message to tianon asking him if he remembers what the original numbers were. He told me long ago, I think it was around 1000. This was well before the go shim or dockerd existed.
I guess you don't have anything CPU intensive running.
General good practice for sysadmins is to affine CPU cores/threads so that you don't end up flushing the CPU cache too aggressively and you don't split your VM's over NUMA zones because memory locality is important.
Well, I was asking what the actual result was. GP (who didn't respond to me) mentioned it was an experiment.
TheDong gave experimental results in a reply though.
The server hosting the article above is also running 38 other containers - databases, WordPress blogs, Jenkins (CI), Gogs (Git), etc. And it only has 4 GB of RAM.
If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured.
If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few
The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container.
"containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most.
The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k.
Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total.
Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu
This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.