[kosma]

Until a few days ago I was a DevOps engineer at a medium-sized tech company. My primary responsibility was dockerizing their applications and infrastructure.

I quit the job.

The scenario played out just as you said: I ended up single-handedly and poorly re-engineering something that already existed (they did have a working Ansible setup) for no visible gain. "Swimming in containers upon containers" is exactly what happened; they kinda worked, but the farther we got, the more kludges piled on top of each other. In four months work we didn't even hit production - the most we got was a CI/QA service that was actually nothing more than a loose bunch of Python scripts. Between managing dev/test/prod differences, tracing missing logs, removing unused volumes, networking all that stuff together and trying to provide at least a decent level of security, I realized that I'm wasting everyone's time and money. Developers hated it because it filled their workflows with traps and obstacles. Admins hated it because of the lack of tooling. Business hated it because it caused unexplainable delays. The only thing we really accomplished was some compliance with the The Twelve-Factor App - something that could've been done in a week. Hardly a victory.

My advice? Forget about Docker unless your primary business is building hosting systems. It will take years before Docker gets mature enough for production, and not without a ton of tooling on top of it and some major architectural changes. Until then, go back to the old UNIX ways of doing things... it worked perfectly since the Epoch and it will continue to work long after the 32-bit time_t rolls over. You'll be fine.

[Nyetan]

Bother you for a little advice? I'm working at a mid-sized tech company and am evaluating Docker for CI, testing and limited, internal deployment usages.

The services in question are built with a hodge-podge of shell scripts and build tools, so getting them all to compile locally is a challenge, let alone deploying them. My hope was that containerizing the builds would isolate any configuration problems, and that containerizing the deployed services would cut down on outages by permitting trivial rollbacks (say, by snapshotting all the service containers before each deploy and merely restoring them should a deployment fail). Of course, all of the above could be fixed by traditional means (e.g. rewriting the build system with a single, standard tool; streamlining the deployment process, etc.), but it seemed like Docker could solve 80% of the problems while easing the implementation of the proper solutions down the line.

Considering the above, do you still think Docker's a poor fit for business that aren't building hosting systems? Oh, and any nuggets of wisdom you could throw to a newcomer to the industry? :)

[boundlessdreamz]

I think you will get more bang for the buck by using something like ansible or salt (I have only used ansible and love it and I have heard salt is comparable)

You don't have a standard repeatable way to set up an environment now. You need to do that first before jumping on docker I think. Once you have that, you can start replacing parts of the setup with docker and see if it fits your needs.

The advantage of ansible is that it is idempotent and the changes it makes to the system are the same ones you make manually or via bash scripts. So it is quite easy to debug

[kosma]

Seconded. The most important feat you need to accomplish is being able to spin up new instances/environments on a whim. Once you have that, provisioning, testing, failover, scaling and high availability get much easier. Vagrant is decent in this role; the only caveat is that the majority of developers tend to never reset their environment and let the cruft accumulate. It's more of a people problem than a tech challenge, though: if your company culture is "if it works, don't touch it", then you have way bigger problems than choosing the right virtualization solution.

[Nyetan]

Aah, brilliant -- and they even have a book :D thanks for the recommendation!

[xorcist]

The big pro with Ansible and similar tools is that the scripts are actually very readable, with clear best practices.

If you come to a new place and "there's an error in here somewhere", the difference between layers of images held together with shell scripts and a Ansible/Puppet/Chef script is like night and day.

[pnathan]

Contra most of the trend on this thread, I am super pro-Docker (I'm actually surprised so many people are unhappy with it - it seems to be clearly head and shoulders over other systems). I would argue that in your situation you need to use docker as your build system at the least.

Something like the following

    docker run -v `pwd`:/tmp/buildresult your-weird-hodpodge build-command

Among other things, I see docker as an extremely useful mechanism to decouple server maintenance (build server and deployment server) from the tool maintenance. It can dramatically help reproducibility, etc.

[kosma]

The moment you do

    docker run -v `pwd`:/...

you're no longer running pure Docker, but Docker + a shell script. Those shell scripts bloat horribly, are not portable, and are a pain to maintain. This is precisely my problem.

[pnathan]

The strong pressure I'm laying on people in my company is that Check Your Stuff In, so there might be a run-docker.sh that's checked in, which then can be reviewed and evaluated. It's not per se ideal, but it's a sight better than a nest of Jenkins scripts outside of source control.

[vacri]

Make your build servers match your production environments. Deploying on SuprCoolOS v4? Build on the same. If it then doesn't build on your buildservers, it's a problem for the developers. If your developer can only get it to build on their UltraCoolOS v9 laptop, it's up to them to fix it - after all, their job is to write stuff that works in production, not on their laptop. They can run a VM or change their OS or whatever. Or ask for a change to production :)

As for rollbacks, the exceptionally bad Docker tagging system just adds headaches to rolling back efficiently. If your production OS has a package management system, consider building packages for that - after all, it will have been battle-tested and known to work on that OS. There will be a learning curve for any packaging system, but using a native one means less faffing around later - remember also that docker is changing a lot with each release.

Also, as mentioned in the article, logging with docker is difficult and hasn't been solved properly yet, and if you like production logs for troubleshooting, Docker requires some attention before you can get those logs. My devs just run the app and watch STDOUT... which isn't easy to log in docker. Then, of course, they complain that they don't have production logs to debug, and subsequently complain when I ask them to modify their logging so I can slurp it :)

Anyway, Docker is not a packaging system for use in-house; if you're only using it to package stuff... you will be ripping it out later on down the line (this is what happened to me). On the other hand, if you open-source your stuff and want to provide 'canned images' for random members of the public to use, then there is a point to using docker, since you don't control what those host machines will be running.

In short, Docker is a complex ecosystem with it's own learning curve, and doesn't really save you from learning curves for other things. If you can't articulate the exact problems that using Docker will solve for you in production, I would advise against it.

Edit: If you need a standardised provisioning system, start out with Ansible. It's pretty straight-forward. Admittedly I've only used it and Puppet... and puppet is better aimed at large/complex infrastructure environments.

[catern]

You don't need Docker to make a standard build environment. You don't need "chroot on steroids", you just need chroot.

[Nyetan]

I certainly could (and actually hadn't thought of doing that; the hype was really getting to me)! Part of the difficulty is in fixing the setup process, though, as over the years this system seems to have accrued a fair amount of -- shall we say -- character. Jumping in and fixing it up to the point of making environment setup a short, portable and error-free process (such as (1) pull a vm image (2) feed it a commit hash to build, test and deploy) would optimistically take weeks. This drove my desire to short circuit the entire affair and stick it all into a VM. That's what I was alluding to vis. proper solutions :)

[rmgraham]

I've found docker to be "chroot with all your custom fs layouts and custom mount scripts combined" - in other words a much more convenient chroot.

[xofer]

True. Docker with a "ton of tooling on top of it" is going to be a VM with versioning, which as discussed, has its own set of issues.

[chasb]

We run Docker in production, as a PaaS no less, and this reads like a list of reasons to use a new-generation PaaS.

[kordless]

Thanks for the chocolate again! :)