Anyone know how to get smaller docker images? I thought if I had all the previous layers in the docker registry that an upload would just be the size of the diff of the new layer, but this seems to never work.
Some docker registries isolate the layer cache per account to prevent cache poisoning attacks and data leaks. This means you might only take advantage of the registry caching if you have already pushed the first version of a tagged image.
If you want to get extremely small docker images you might also want to take a look at Google's distroless images and using mutlistage builds.
GCR boasts it has a global layer cache. Also, this is a thing that should only ever effect your very first push. You should only be seeing it twice if:
1. None of your image layers are the same between builds (ADD as one of the first instructions for instance)
2. You are distributing your code to many people and they are building them into entirely separate accounts. (you send me your code and I build, tag, and push it to my dockerhub account).
Unfortunately, choice 2 sounds like us. I was looking for some way to short-circuit that (by maybe shipping the repo already loaded) as the product runs in the customer's cloud, and the images are built on their machines.
If that's the case you could give them a pre-built copy of your containers (assuming your builds take a very long time this might be worth it).
There's two commands: `docker save` and `docker load`. It tars the history, layers, etc into a single file. You can further compress it for distribution. I've had a lot of luck with it.
Your client would then download your source, `docker load` your prebuilt copies to warm their cache, make their modifications, and further builds would be much faster.
They'd still pay that first penalty for pushing to their internal registry but that shouldn't take too long since that's essentially just a file copy.
If you can get all of your images on the same machine to use the same base, you’ll get somewhere. Being careful of layer order, using a script to run the most disk-intensive layers also helps.
I’m having okay luck with alpine base images right now, but app versions are less flexible.
If you want to get extremely small docker images you might also want to take a look at Google's distroless images and using mutlistage builds.