[tobbyb]

Docker was based on LXC and there are some important differences that should be more well known.

The main difference is LXC boots an init in the namespace as the first process so you get a standard OS environment with multi-process support. Docker runs the application directly so you get a single process environment, unless you decide to run a process manager. IMHO this creates far more problems than it solves as anything nonstandard would, as you now need workarounds for basics like logging, networking, daemons and anything that expects a standard OS environment

Docker also uses layers to build containers and ephemeral data. None of these 3 critical decisions have seen much technical scrutiny resulting in persistent confusion in the ecosystem and with scrutiny many users will find they can add far more management overhead and complexity to containers. We have written about some of these issues here. [1]

For instance OP is referring to unprivileged containers and user namespaces that let you run containers as a user process. LXC has supported these since 2013. Docker added user namespaces support in recent versions. However taking just one issue - because of the use of layers to build containers you can't use unprivileged containers for app builds as only root users can mount filesystems. So a lot of these decisions have all kinds of tradeoffs that should be better known and discussed. [2]

Disclaimer: Founder at Flockport and trying to build a simpler alternative with LXC with support for app builds, provisioning, orchestration, service discovery and an app store. [3]

[1] https://www.flockport.com/guides/say-yes-to-containers.html

[2] https://www.flockport.com/guides/understanding-layers.html

[3] https://www.flockport.com/guides/cluster-services

[cpuguy83]

User namespace support for Docker has been in for years now, not just "recent versions".

Not sure what you mean by "app builds" here. The container process is in a user namespace, not docker itself and "docker build" will work just fine. Maybe you are referring to something else? Let me know, would love to take a look.

Layers are largely an implementation detail in Docker. Yeah they are there, you can even check the sha of them, but not something users interact with. I don't think I've personally heard any of tbis confusion except for very old versions of Docker before content-addressabity and the concept of layers did leak through (each layer was an "image")... maybe there's still some lasting confusion here? I could see this I suppose with things like older books/blogs/etc referring to them as a first class citizen. They are definitely not and certainly should not be. One of the early design mistakes which are long since corrected (docker 1.10 introduced content addressable images, and the concept of the image as a whole rather than an image being a tree of other images).

Also, a side note, "docker run --init" gets you the init you are looking for, can set it as default in the daemon as well.

Note, I've been a maintainer/contributor to Docker for 5 years now.

[cyphar]

Hiya Brian!

> Layers are largely an implementation detail in Docker.

You say this, but they are so deeply baked into many tools that there have been many hundreds of talks about how to "reduce the size of your Docker image with this one simple trick!" which reduces down to understanding how layers in Docker images work. Layers definitely leak through a lot of the tooling.

I'm hoping that my OCIv2 proposal will remove the practical need for layers entirely -- and also remove a lot of the cargo-culting which has popped up around "how to get smaller images". Maybe Docker might adopt this in the future (after you guys get OCIv1 support :P).

> Also, a side note, "docker run --init" gets you the init you are looking for, can set it as default in the daemon as well.

This is different to LXC. With LXC, the "init" is actually systemd, not just a zombie reaper -- and when you do an "lxc attach" you are actually put into a new console (getty and all).

Yes, you can run (after some headaches) systemd inside Docker, but LXC definitely handles this much better because that's the usecase it was designed around. It's not really a defect in Docker, it's just not what Docker was designed around.

[cpuguy83]

> "reduce the size of your Docker image with this one simple trick!"

Truth!

> I'm hoping that my OCIv2 proposal

I've seen (and largely agree with) your post (somewhere? Can't remember where) here, are you working on a formal proposal?

> "init" is actually systemd

Fair enough. I think this opens up additional security issues (requires SYS_ADMIN as I recall), though?

[cyphar]

> I've seen (and largely agree with) your post (somewhere? Can't remember where) here, are you working on a formal proposal?

I'm currently working on a blog post to describe and justify the design, as well as a PoC. The formal proposal for OCI will come after the PoC (and after sufficient benchmarking and discussion within the community). But I plan to write a spec document along the PoC.

[tobbyb]

Layers are integral in Docker, layers were pushed as enabling reuse, composibility, 'immutability', every 'run' statement is a layer. They are not an implementation detail. Managing all these layers adds complexity and impact users from choosing storage drivers, to using layers to building images, and if you are going to use layers you need to be a privileged user to mount all these layers. [1]

Here is an example. I build an Nginx container with no layers from a base alpine image. Users can download and run it in a layer so the original Nginx container remains untouched or run a copy of it. If there is a security alert or update I rerun the build. No layers used. You build the same Nginx container will multiple layers, and have all the overhead of tracking and managing these layers. And users then download the Nginx container made up of all these layers and run it. If there is a security update in any lower layers it needs to rebuild. You have used a workflow and designed an entire system based on stacking layers to compose and build your image. That is a lot of overhead and complexity in basic image management. What is the benefit?

[1] https://kinvolk.io/blog/2018/04/towards-unprivileged-contain...

[cpuguy83]

Right, they are integral but the user doesn't interact with them and they can be swapped out for a different implementation and the user would never know outside of noticing the impact that the change has (or trade-offs such a change would make).

Caching for `docker build` really doesn't have anything to do with layers except as a means of storage, it doesn't need to be layers at all.

To be honest, layers aren't even all that efficient (except in perfectly ideal scenarios), just better than no sharing at all and no one has taken the time to come up with something better yet...because frankly there are many other things that need improvements before bothering with it.

Again, just figured I'd mention again... since you are trying to explain Docker to me... I'm a Docker maintainer and have been for the last ~5 years. Happy to chat about this.

[linuxftw]

Layers may or may not benefit any particular org. If you build a lot of your own containers from the same base layer image (microserver1-container-a .. microserver2-container-N), then you could save a lot of bandwidth/storage pulling the base layer once. Naturally, if you build everything from alpine:latest, then a large majority of your containers are going to have difference base layers and you're going to lose those savings.

[vetrom]

Similar to what linuxftw notes, when Docker hit the scene it was basically competing with the likes of Heroku where you have a few common runtime containers on a shared runtime system. When trying to implement one of those you want to maximize some sorts of shared common components and page/file sharing to create some efficiencies in memory sharing/oversubscription.

[vikrantrathore]

@tobbyb Thanks for all the work on flockport your tutorials on LXC helped me to have initial production systems ready. I applied many of those concepts specially for networking using flockport tutorial.

Although today my system with LXD is much easier and getting better with every subsequent release of LXD, I owe you gratitude for helping me to get off the ground.

[tobbyb]

@vikrant Glad to hear that! Networking in Linux is incredibly feature rich and the LXC team have done a great job on container networking.