The smallest useful container I know of is 129B. It was created to test how many containers docker can spin up while reducing the overhead of what was in the container itself.
tianon/sleeping-beauty latest 2e8193709fa7 6 months ago 129B
If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured.
If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few
The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container.
"containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most.
The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k.
Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total.
Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu
This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.
I didn't hit any limit, but since I don't have swap it would likely be oom kills.
The first part of my post is speculating, the second part I ran an arbitrary number and observed resource usage to allow extrapolation, but didn't hit a limit.
Thanks for reproducing the experiment. :) I sent a message to tianon asking him if he remembers what the original numbers were. He told me long ago, I think it was around 1000. This was well before the go shim or dockerd existed.
I guess you don't have anything CPU intensive running.
General good practice for sysadmins is to affine CPU cores/threads so that you don't end up flushing the CPU cache too aggressively and you don't split your VM's over NUMA zones because memory locality is important.
Well, I was asking what the actual result was. GP (who didn't respond to me) mentioned it was an experiment.
TheDong gave experimental results in a reply though.
The server hosting the article above is also running 38 other containers - databases, WordPress blogs, Jenkins (CI), Gogs (Git), etc. And it only has 4 GB of RAM.
Another example of a really small container that doesn't do much, but is made without using assembly directly is the Docker "hello-world". It's built from C without linking in libc:
I always thought this was a bit misleading. A "hello world" container is 1 kB, but the bare minimum container that does something useful in practice is rarely less than 100 MB in size.
If you base off alpine, you can get useful containers quite a lot smaller than 100MB.
One example i use is an agent i deploy to kubernetes clusters to do some security scanning. The scripts are ruby and the image clocks in at 9MB compressed https://hub.docker.com/r/raesene/kaa-agent/tags/
If you're into go, it's not too hard to get very small (<5mb) shippables by statically compiling against musl and using upx. Here's a somewhat scrubbed Dockerfile for a gRPC/rest service I use at work: https://gist.github.com/jbergstroem/680cb7db6f90319dcd7666f3...
5mb still sounds like a lot, considering you could squeeze Linux 1.3 on a 1.44 mb floppy with a (compressed) rootfs... I mean does the runtime really do that much more than a full (although old) os kernel and a C library/runtime and apps?
Alpine's package manager has the great property that you don't need to update the index in order to fetch a package IIRC; the whole `apt-get update && apt-get install && <cleanup apt-cache>` dance is quite tedious in debian-based Docker containers.
No, you still need to, but there's a compact syntax for it that will update and discard the index in a single 'add' command. It's unavoidable - somewhere some querying is happening in order map the package name/ver to a download link.
I find the haproxy (alpine) Dockerfile a great example on how to tender to container file-size. It uses the syntax you're referring to, temporary build virtuals (should be multistage today I guess) and static linking: https://github.com/docker-library/haproxy/blob/2d393f2b59824...
> but the bare minimum container that does something useful in practice is rarely less than 100 MB in size.
I’ve made containers using code written in most common programming languages (python/go/ruby/c/rust/heck even PHP/etc) which were easily under 100mb, most significantly less. If your containers are frequently >100mb, I say you’re either using the JVM or are doing it wrong!
Yeah, I can easily make JVM containers under 100mb too usually, but it’s not always that small, so I wanted to give a little benefit of the doubt there.
We already have a portable container format for executables with no dependencies... It's called an ELF binary... Why would you even put a static binary in a container in the first place? I don't get it.
Containers are not just about shipping binaries. In fact, they don't really add too much in that category. Namespace isolation and resource limiting with cgroups are the real benefits.
And in most real world cases, you will still need at least libc and ca-certificates.
I guess if you wanted to take advantage of the implicit cgroup that the process then becomes a part of? Of course you could set up such a cgroup without Docker, but maybe it's just more convenient in certain cases to use Docker for it to keep your deployment process more consistent.
I guess you mean something like Compose or Kubernetes.
I think it's weird these things force you to use the docker runtime, personally. I would really love a more flexible definition in kubernetes of what a "resource to be ran" means.
I know they support multiple container runtimes now, but what if I dont want a container runtime at all?
Nomad supports raw executables to be downloaded and scheduled,
which is nice(https://nomadproject.io) but then again, kubernetes seems miles ahead in what it supports (autoscaling, volume claims, RBAC etc)
Otherwise, more traditional means of managing your services can be employed. I've got a lot of leverage out of systemd myself. Which by the way, supports all the features of a proper container runtime. You can namespace your executable, Chroot it, limit what devices it can access, etc, which is kinda awesome. Check out `man systemd.exec` and `man systemd.resource-control`
Kubernetes absolutely does not force you to use the docker runtime. In fact, there has been a lot of work to avoid this by creating the CRI[0].
Kubernetes also supports extensions like the Third Party Resource or their successor, Custom Resource Definitions. KubeVirt[1] is an example of extending resources to include VMs
This is of course pretty pointless, I mean it's neat but all these tiny containers don't really do anything so it isn't much more than a cool trick.
However there is a great lesson to take from this: you can create single-binary but really useful containers for just a few MBs, which is nothing in practice for a usual sized server, an a lot more lightweight than usual containers based off Alpine or Ubuntu.
In fact I run my static websites using that: a small 8-ish MB statically compiled web server "written" (it's really just library glue) in Go.
I've done this with a couple of applications, using something like fileb0x in the build process to convert the files into Go source which can then be compiled into the final executable.
If you want to reduce the size more, try emitting a single layer as a tarball which you can pipe to docker load. That will reduce your layer count to one, plus you can omit a bunch of the metadata that Docker includes when you build from a Dockerfile (it doesn't seem to care at runtime).
Notice that the executable itself contains less than 100 bytes of instructions, but the file is still 736 bytes. Even without optimising the Asm itself (I can see at least 2-3 bytes improvement at a glance), that could probably be reduced even further:
736B may seem tiny to most people, but if you're working in Asm that's a lot --- there's plenty of interesting things (beyond "call the OS a few times") the demoscene has done with smaller binaries; here's an assortment of 512B ones:
I did a similar thing, using a tiny executable that someone else had made. However, even though the container was around 100 bytes, I couldn't make a container that was smaller than some much larger number, maybe 512kb? This was in 2015, so maybe that limit has changed - any docker folks know anything about this?
OT, what's the smallest windows/.net containers out there? I have some legacy stuff I'd like to dockerize but they end up so big and slow to deploy. I've just been rewriting them.
The reverse proxy is configured to route requests to any container that is currently running. For example, if I have a Jenkins container, as long as it is running, the reverse proxy will send requests to it. In this particular case, I need to proxy a remote host. In order to do that, I have a container running with the appropriate labels, the proxy sees the container running, and it proxies requests to the remote host.
I think I'm being particularly dense here, but do you mean the proxy is proxying to your tiny 1kb container? If so, what happens to the traffic? Or because the tiny container is running the proxy proxies somewhere else? i.e you're using the presence of a container with a label as configuration for the proxy?
I know, it's a bit difficult to explain and a tad confusing. No, the proxy is not sending any traffic to the container. Rather, the container, by simple virtue of being in the running state, tells the proxy to send requests to the domain specified by the label on the container.
Details: http://thebsdbox.co.uk/in-pursuit-of-a-tinier-binary-er/
Code: https://gist.github.com/thebsdbox/29e395299f89b52214b66269f5...