[iamleppert]

I've said it before and I'll say it again: pre-mature infrastructure optimization is the root of all evil.

Do me a favor and if you got a startup, stay clear of all this. Everyone wants to reinvent their own flavor of heroku and make your deployment and build pipeline god-awful complex. Their tool of choice? Docker.

Before you know it you'll be swimming in containers upon containers. Containers will save us, they'll cry! Meanwhile you have 0 rows of data before you've paid them their first month's salary and have spent time on solving problems of scale you'll never have.

Focus on your product, outsource the rest. And leave customized docker setups to mid-stage startups and big corps who already have these problems, or at least the money and people to toil on them. Not everything needs to be a container! And most companies are not and will never be Google!!

[kosma]

Until a few days ago I was a DevOps engineer at a medium-sized tech company. My primary responsibility was dockerizing their applications and infrastructure.

I quit the job.

The scenario played out just as you said: I ended up single-handedly and poorly re-engineering something that already existed (they did have a working Ansible setup) for no visible gain. "Swimming in containers upon containers" is exactly what happened; they kinda worked, but the farther we got, the more kludges piled on top of each other. In four months work we didn't even hit production - the most we got was a CI/QA service that was actually nothing more than a loose bunch of Python scripts. Between managing dev/test/prod differences, tracing missing logs, removing unused volumes, networking all that stuff together and trying to provide at least a decent level of security, I realized that I'm wasting everyone's time and money. Developers hated it because it filled their workflows with traps and obstacles. Admins hated it because of the lack of tooling. Business hated it because it caused unexplainable delays. The only thing we really accomplished was some compliance with the The Twelve-Factor App - something that could've been done in a week. Hardly a victory.

My advice? Forget about Docker unless your primary business is building hosting systems. It will take years before Docker gets mature enough for production, and not without a ton of tooling on top of it and some major architectural changes. Until then, go back to the old UNIX ways of doing things... it worked perfectly since the Epoch and it will continue to work long after the 32-bit time_t rolls over. You'll be fine.

[Nyetan]

Bother you for a little advice? I'm working at a mid-sized tech company and am evaluating Docker for CI, testing and limited, internal deployment usages.

The services in question are built with a hodge-podge of shell scripts and build tools, so getting them all to compile locally is a challenge, let alone deploying them. My hope was that containerizing the builds would isolate any configuration problems, and that containerizing the deployed services would cut down on outages by permitting trivial rollbacks (say, by snapshotting all the service containers before each deploy and merely restoring them should a deployment fail). Of course, all of the above could be fixed by traditional means (e.g. rewriting the build system with a single, standard tool; streamlining the deployment process, etc.), but it seemed like Docker could solve 80% of the problems while easing the implementation of the proper solutions down the line.

Considering the above, do you still think Docker's a poor fit for business that aren't building hosting systems? Oh, and any nuggets of wisdom you could throw to a newcomer to the industry? :)

[boundlessdreamz]

I think you will get more bang for the buck by using something like ansible or salt (I have only used ansible and love it and I have heard salt is comparable)

You don't have a standard repeatable way to set up an environment now. You need to do that first before jumping on docker I think. Once you have that, you can start replacing parts of the setup with docker and see if it fits your needs.

The advantage of ansible is that it is idempotent and the changes it makes to the system are the same ones you make manually or via bash scripts. So it is quite easy to debug

[kosma]

Seconded. The most important feat you need to accomplish is being able to spin up new instances/environments on a whim. Once you have that, provisioning, testing, failover, scaling and high availability get much easier. Vagrant is decent in this role; the only caveat is that the majority of developers tend to never reset their environment and let the cruft accumulate. It's more of a people problem than a tech challenge, though: if your company culture is "if it works, don't touch it", then you have way bigger problems than choosing the right virtualization solution.

[Nyetan]

Aah, brilliant -- and they even have a book :D thanks for the recommendation!

[xorcist]

The big pro with Ansible and similar tools is that the scripts are actually very readable, with clear best practices.

If you come to a new place and "there's an error in here somewhere", the difference between layers of images held together with shell scripts and a Ansible/Puppet/Chef script is like night and day.

[pnathan]

Contra most of the trend on this thread, I am super pro-Docker (I'm actually surprised so many people are unhappy with it - it seems to be clearly head and shoulders over other systems). I would argue that in your situation you need to use docker as your build system at the least.

Something like the following

    docker run -v `pwd`:/tmp/buildresult your-weird-hodpodge build-command

Among other things, I see docker as an extremely useful mechanism to decouple server maintenance (build server and deployment server) from the tool maintenance. It can dramatically help reproducibility, etc.

[kosma]

The moment you do

    docker run -v `pwd`:/...

you're no longer running pure Docker, but Docker + a shell script. Those shell scripts bloat horribly, are not portable, and are a pain to maintain. This is precisely my problem.

[pnathan]

The strong pressure I'm laying on people in my company is that Check Your Stuff In, so there might be a run-docker.sh that's checked in, which then can be reviewed and evaluated. It's not per se ideal, but it's a sight better than a nest of Jenkins scripts outside of source control.

[vacri]

Make your build servers match your production environments. Deploying on SuprCoolOS v4? Build on the same. If it then doesn't build on your buildservers, it's a problem for the developers. If your developer can only get it to build on their UltraCoolOS v9 laptop, it's up to them to fix it - after all, their job is to write stuff that works in production, not on their laptop. They can run a VM or change their OS or whatever. Or ask for a change to production :)

As for rollbacks, the exceptionally bad Docker tagging system just adds headaches to rolling back efficiently. If your production OS has a package management system, consider building packages for that - after all, it will have been battle-tested and known to work on that OS. There will be a learning curve for any packaging system, but using a native one means less faffing around later - remember also that docker is changing a lot with each release.

Also, as mentioned in the article, logging with docker is difficult and hasn't been solved properly yet, and if you like production logs for troubleshooting, Docker requires some attention before you can get those logs. My devs just run the app and watch STDOUT... which isn't easy to log in docker. Then, of course, they complain that they don't have production logs to debug, and subsequently complain when I ask them to modify their logging so I can slurp it :)

Anyway, Docker is not a packaging system for use in-house; if you're only using it to package stuff... you will be ripping it out later on down the line (this is what happened to me). On the other hand, if you open-source your stuff and want to provide 'canned images' for random members of the public to use, then there is a point to using docker, since you don't control what those host machines will be running.

In short, Docker is a complex ecosystem with it's own learning curve, and doesn't really save you from learning curves for other things. If you can't articulate the exact problems that using Docker will solve for you in production, I would advise against it.

Edit: If you need a standardised provisioning system, start out with Ansible. It's pretty straight-forward. Admittedly I've only used it and Puppet... and puppet is better aimed at large/complex infrastructure environments.

[catern]

You don't need Docker to make a standard build environment. You don't need "chroot on steroids", you just need chroot.

[Nyetan]

I certainly could (and actually hadn't thought of doing that; the hype was really getting to me)! Part of the difficulty is in fixing the setup process, though, as over the years this system seems to have accrued a fair amount of -- shall we say -- character. Jumping in and fixing it up to the point of making environment setup a short, portable and error-free process (such as (1) pull a vm image (2) feed it a commit hash to build, test and deploy) would optimistically take weeks. This drove my desire to short circuit the entire affair and stick it all into a VM. That's what I was alluding to vis. proper solutions :)

[rmgraham]

I've found docker to be "chroot with all your custom fs layouts and custom mount scripts combined" - in other words a much more convenient chroot.

[xofer]

True. Docker with a "ton of tooling on top of it" is going to be a VM with versioning, which as discussed, has its own set of issues.

[chasb]

We run Docker in production, as a PaaS no less, and this reads like a list of reasons to use a new-generation PaaS.

[kordless]

Thanks for the chocolate again! :)

[stephenr]

I'm definitely no fan of Docker or the ridiculous containerisation trend, but I think I may disagree with one key thing you said:

> Focus on your product, outsource the rest.

What do you mean by outsource the rest?

Do you mean, "hey we're using AWS <Everything>-as-Service because we don't want to manage a DB cluster or deal with a load balancer?

Or do you mean, rely on existing available tools and stop reinventing the wheel every week?

[statico]

> What do you mean by outsource the rest?

iamleppert means: Identify your company's core competency and do that in house, but outsource or avoid that which is not your core.

For example, we're making a game. Gameplay, art, and tech is all done in-house and not with remote contractors because it needs to be -- it's the part of the product we love and the part our players will end up loving. Email, forums, chat, HR, applicant tracking systems, and git hosting are outside of our core and best handled by others.

[Gigablah]

But if you're making a game, and create your own chat service instead, you end up as Slack.

[statico]

We in fact started with an HTML5 group chat application very much like Slack but ended up making a game.

[stephenr]

Define "handled by others".

Installing an exchange server is arguably letting email be "handled by others" because you are not responsible for how it works, just the setup and monitoring, which could be handled by in-house staff or by a contractor.

My point is that "focus on the core competency" doesn't have to mean "make our company reliant on a dozen other SaaS businesses who may go offline or change their business model/functionality on a whim"

[kordless]

There are reasons other than scale for containerizing your application. Continuous deployments become a bit easier, for example. Reliability, especially during rolling updates, is improved. It's easier to build a zero downtime app that is in continuous deployment with a containerized stack than it is with more traditional instances.

Regarding the outsourcing, that's what we're shooting for at Giant Swarm. We've written a stack that runs containers and manages the metal underneath for you. We run the solution as a shared public cluster at giantswarm.io, but can also do private hosted deployments or managed on-prem deployments. It's a complete solution for running containers that feels like a PaaS, but without all the opinionated crap associated with a PaaS.

We're basically offering to be your little devops team that could - with containers.

[jordanthoms]

I do agree with you, but we're on Heroku and that also has it's downsides - the biggest one we've found is that CPU performance on 1x and 2x dynos is very unpredictable - We've had to move to Performance dynos ($500/month each!) to get decent performance and prevent random timeouts.

Services like Cloud66 are interesting (they manage deployments onto your own EC2 or other cloud infrastructure), but the developer experience doesn't quite match Heroku yet.

Heroku really needs some more competition...

[theatraine]

Azure App Service (https://azure.microsoft.com/en-us/services/app-service/) is Azure's Heroku competitor. It's quite close in features and ease-of-use. Disclosure: I work for this team.

[jordanthoms]

That looks quite nice in the few minutes I spent looking at it. The lack of Ruby support is a issue though, and we're also all on OSX and prefer CLI interfaces for most stuff.

Also, one of the killer features of Heroku (which few services seem to replicate) is log drains - I can easily add a http or syslog endpoint and have Heroku send the logs over. The other killer feature which isn't often replicated is One-off dynos, where we can spin up a new instance and get a console attached to it in one command - useful for running database migrations or using Ruby as a CLI to access data.

If we were on .NET that would probably be attractive, but it's still not really competing with Heroku.

[agiamas]

you summed up the reasons I use Heroku myself, gz.