[hinkley]

Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build.

Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.

I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.

[watermelon0]

You do know that spinning up a new VM only takes a few seconds? With projects like https://firecracker-microvm.github.io/, the difference between launching a new Docker container or a new VM is negligible.

This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.

[hinkley]

The cost is not spinning up the vm, it’s maintaining the images. Docker composability reduces the combinatorics problem to a dull roar, and democratizes some of the maintenance effort. You want an image with the bug fix from the latest point release of python? And you need it by noon? Knock yourself out.

Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.

[ryan29]

The other option that works really well in a single user environment is to bind to the runner's Docker daemon. That way builds run as siblings of the runner's daemon rather than as children via docker-in-docker.

The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).

If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).

[hinkley]

For us it’s a matter of the CI tool fetching the source code for the docker image, then running docker build, and not necessarily immediately. So you have ‘docker build’ happening toward the end of a set of other tasks. Which I’d really like to have running on a fresh VM or container.

You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.