[cgb_]

    This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of "trust" and "upgrades".

Prebuild VMs? Sure, I wouldn't touch them except for evaluating a project, and for commercial software you may not have a choice.

But docker containers at least usually provide a dockerfile that describes exactly how a binary image is built. You just clone the source repo, audit the few lines of build commands and then build your own private registry. It's nearly no more trust than following the instructions of README or INSTALL. Just because fools are pulling down pre-build images and running in their datacentre, doesn't mean that's what way you should do it. And the problem with 'old-school' sysadmins is they are often far too quick to reject new practices, citing tired excuses based on misunderstandings of the technologies.

    Ever tried to security update a container?

Yeah I have. It's easy if you have already built your 'stack' to scale horizontally (which means you have at least 2 or more of everything in a HA or LB config). You rebuild against a fully patched base-OS container, spin-up, send some test load to it & validate, then bring into service. Repeat for rest of nodes at that tier.

If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...

[vacri]

then you are doing containers wrong...

The old-school sysadmins I know scoff at Docker's idea of 'containers'. Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. To them, containers are a way of enveloping a process to limit it, not a way of distributing packaged software. They may or may not be doing 'docker' right, but they certainly know what 'linux containers' are.

[cgb_]

Well I should have qualified it with 'docker containers'. But yeah, those that have been around long enough in Linux container land have all dealt with vserver, openvz, lxc, etc, and all of those carried around this 'entire copy' of an OS, per container (ignoring vserver's vhashify). Docker helps you to spin up N containers running all sorts of applications based on the single master image.

Docker, whether your view is good or bad, brings something more than just another container implementation to the table...

[eropple]

> Linux containers were already a thing, and don't need an entire copy of an OS ported around with them.

Neither do Docker containers. You can build off scratch and put the literal bare minimum you need in it. I've done it a few different times. It's rarely done because the time and effort almost never makes up for the complexity and cost, but if your old-school sysadmins are scoffing it's on them.

[vacri]

Response to exactly that idea from one of these guys: "Why do you need docker to just build an executable?"

[Torgo]

Deterministic builds. Isolating shitty build scripts from my system.

[eropple]

I don't. But it helps make a straightforward deployable of highly coupled libraries and tools in a way that's more comprehensible to other people.

But I'll get off your lawn now.

[vacri]

Ain't my lawn; I consider myself a mid-range sysadmin. I stepped out of support and into sysadmin land about 4 years ago. But I know some 'from-the-birth-of-linux' guys, who live and breathe this stuff in a way I never will. When I get home from staring at terminals, I want to watch movies and play video games, not swear at something on a breadboard :)

[peterwwillis]

> If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...

A container is a chroot environment for running a service. Basically. It's perfectly possible to 'yum upgrade' or 'apt-get upgrade' them.

I think what you're trying to say is this is a bad idea because containers are not supposed to be managed individually. The magical fluffy dream of Docker is to never have to manage another individual OS again; just make one image, then kick it out to all your machines and make them use the new image. No resolving dependencies, no verifying individual file checksums, no upgrading or downgrading or conflicting different package versions for every server and service. Just do it once, and then push it out everywhere, and everything magically works.

Right?

Here's the thing: Containers don't do away with the idea that a developer might use a totally bleeding-edge piece of software to update one of your many apps, and that you may end up with multiple incompatible versions of software on your systems. In fact, Docker kind of trades on that as a feature. "Install any shit software you want and it'll never conflict with other containers!" But this lie is shown for what it is once you start looking at containers as individual physical machines.

Back in the day we had to 'yum install some-specific-architecture-and-version-of-this-package' on a particular machine to make that machine serve that software. You do the same with Docker, because you have a particular Docker container with a specific version of the package, and all the other packages and OS requirements in that container. You still have a one-off machine to install, maintain and troubleshoot. The only differences are it isn't physical anymore, and you perform updates on the image, not on the machine.

Just like you ended up with a machine (or three machines) with three different versions of BDB, you end up with three containers with three different versions of BDB. Instead of using 'apt' or 'yum' to install them, you write their Dockerfiles and build them, test them, then roll out their updated images. You do a lot more from scratch now because a Linux distro hasn't done the work for you, and so you also run into all the headaches that someone packaging software for a Linux distro usually runs into.

One of the worst things about the 'new devops way' seems to be the non-reproducibility of things like Dockerfiles. To build a Docker image we slap together whatever bleeding-edge files we had on date X-Y-Z, and whatever comes out at the end is considered 'production'. Ignoring the patches, the quality control, the stable released software distributed by distributions on reliable mirrors, and generally without tuning the software at all for the particular system you're running it on. Try to build that Dockerfile again in a year and suddenly it doesn't work the same as it used to, or a bug magically appears on your production system, or you have to apply a patch and now you need to unroll all the commands used for individual stages of the build process for a single package and figure out how to make it still work like the original was built, etc.

Containers by themselves are fine things to use. The problem is how they've effectively sold a lie to everyone that uses them: that there is no sysadmin work to be done. That don't worry, Mr. Javascript Dev, you too can build infrastructure and deploy it without learning the many lessons and best practices of an industry that has been here longer than you've been alive.

This wouldn't even be far from the truth if it was, say, a RedHat-built set of container images, or a Debian-built set of container images. Then at least there'd be an expert who's building software in a reliable uniform way and along a particular standard. They would provide you the software updates, and even provide you with tools and instructions on how to use them to manage your whole software infrastructure without having to write software yourself.

In the age of 'everyone should learn to code', everything can be fixed by writing more code, and copy-and-pasting binaries built on a developer's desktop counts as production deployment. [To be fair, developers were doing this 10 years ago with Java apps, and we hated it]

[cgb_]

I corrected myself for the other person to mean '[..] upgrade your docker containers'. I heavily use LXC containers (and had used openvz and vserver before that) and treat them as individual servers.

All your points about bad sysadmin practices are OS & container agnostic - they can happen on any platform, don't drag Docker into it. Sure there is a culture of 'docker run somebinaryimage [..]' but those people are the ones that do "curl | sudo bash" as well.

Your claim about non-reproducibility of Dockerfiles is bogus. The result of a Dockerfile build gives you precisely the reproducibility you desire. Every time you run a container from that image built from a Dockerfile, you'll get the same filesystem & environment.

Docker 1.6's "Content Addressable Image Identifiers" addresses your build in a year concern by allowing dockerfiles to refer to a digest to ensure you are building against exactly the image you expect (rather than the result of some build process that yum -y upgrades etc, which I think is what you were getting at).

[zobzu]

docker can totally be dragged into this. they keep selling the lie and encouraging most terrible practices and design. talk about self inflicted...

[peterwwillis]

> Every time you run a container from that image built from a Dockerfile, you'll get the same filesystem & environment

The image (and thus container) are the same. Trying to rebuild it from scratch leaves it not the same, typically because the way people put together Dockerfiles and build images does not follow a standard. And it has to do with the container culture.

Let's take a Dockerfile for the CentOS 7 version of nginx (https://github.com/CentOS/CentOS-Dockerfiles/blob/master/ngi...)

  FROM centos:centos7
  MAINTAINER The CentOS Project <[e-mail]>
  
  RUN yum -y update; yum clean all

Right off the bat I think: what the hell? Why are they doing a yum update? A yum update today may very well leave the system in a completely different state than a year ago. There's likely been some package updated in that time, which changes the state of the system. Right off the bat we're screwed.

  RUN yum -y install epel-release tar ; yum clean all
  RUN yum -y install nginx ; yum clean all

And what the hell is this?! There's no version, no build, no checksum. What the hell did we just install? If those packages change in a year, we're screwed. Not to mention 'epel-release' and 'tar' should be set as dependencies somewhere, and the type of dependencies too.

  ADD nginx.conf /etc/nginx/nginx.conf
  RUN echo "daemon off;" >> /etc/nginx/nginx.conf

Oh, cool, just use whatever the hell this config is which may or may not be different from the one that shipped with the original package. And let's just modify it for no apparent reason, too. And definitely make sure we have no way to know what version of that file we're using for this image. Lovely.

  RUN curl https://git.centos.org/sources/httpd/c7/acf5cccf4afaecf3afeb18c50ae59fd5c6504910 \
      | tar -xz -C /usr/share/nginx/html \
      --strip-components=1

The hell? We're pulling some random git sources from a git server which i'm willing to bet has no standard mirrors? On closer inspection it doesn't really look like a git server, but a host named git which hosts files whose names are a checksum, though we don't know where this is from or what exactly it refers to. If this server or file disappears, good luck knowing what in hell was being downloaded here.

  RUN sed -i -e 's/Apache/nginx/g' -e '/apache_pb.gif/d' \ 
      /usr/share/nginx/html/index.html
  
  EXPOSE 80
  
  CMD [ "/usr/sbin/nginx" ]