> And we'd love to have build in support (or at least a common standard or best practices) for orchestration.
Look into BOSH[0][1]. It's a IaaS orchestrator that works for multiple cloud backends -- AWS, openstack, warden and vsphere out of the box. I use it in my day job.
It's already been applied to working with Docker containers.[2]
My problem with every single one of the orchestration solutions I've seen is that they tend to be overcomplicated for small deployments, and make a lot of decisions that I often don't agree with.
Looking at BOSH, my immediate reaction is that they have a huge page just explaining terminology [1]. The fact that they need one (and they do, judging from what's on it) is already a red flag for me.
We run "only" about 150 containers at the moment, so maybe I'd appreciate it more if we had thousands, but to me it seems horribly overcomplicated. And I have an immediate dislike to anything that requires constantly running processes in every VM. That may be necessary for full VM's, but it's one of the reasons I don't think VM focused orchestration solutions are a good fit for container based systems. Our existing (homegrown, on top of OpenVz) system makes heavy use of the fact that every resource in the containers are accessible from the host, and that's one of the thing I like abut Docker too.
It's complicated because orchestration is complicated. You're looking at essential complexity that can't be made to go away.
What you can do is wrangle the complexity into a repeatable, declarative form. That's what BOSH and other orchestration tools do.
Everything on that terminology page is what has been found, in various systems of various sizes, to be necessary to ensure some semblance of robustness and operability as you scale from a single box to thousands of VMs and containers.
In my dayjob I've written BOSH release and deploy manifests that stand up a virtual machine containing 2 containers. I've used other deploy manifests that start a virtual machine with a small cloud of containers. My employers use BOSH to orchestrate multiple clouds on multiple backends with thousands of VMs.
Running a process inside the box is necessary to be agnostic of the substrate. BOSH can't assume that you'll run everything in Docker, or AWS, or vSphere, or any other such system. You can run heterogenous combinations per your requirements. Again, some people absolutely require that capability.
Looking at BOSH, my immediate reaction is that they have a huge page just explaining terminology [1]. The fact that they need one (and they do, judging from what's on it) is already a red flag for me.
We run "only" about 150 containers at the moment, so maybe I'd appreciate it more if we had thousands, but to me it seems horribly overcomplicated. And I have an immediate dislike to anything that requires constantly running processes in every VM. That may be necessary for full VM's, but it's one of the reasons I don't think VM focused orchestration solutions are a good fit for container based systems. Our existing (homegrown, on top of OpenVz) system makes heavy use of the fact that every resource in the containers are accessible from the host, and that's one of the thing I like abut Docker too.
[1] http://docs.cloudfoundry.org/bosh/terminology.html