Hacker News new | ask | show | jobs
by curun1r 3797 days ago
Speaking as someone who rolled his own version of this, there were a lot of more complete solutions out there, but they all involved some technology that I felt would cause more pain down the road. Whether it's Chef/Ansible/Puppet which are popular, but seem targeted at mutable infrastructure (one of our explicit goals was immutable infrastructure) or Mesos/Kubernetes/ECS/CoreOS which seem targeted at a larger fleet of instances than we're running, there didn't seem to be any starting point beyond composing the right set of tools and writing the glue that made sense for us.

What we ended up with uses Terraform for provisioning instances, Docker (and a private registry) for distributing our application code, Consul for coordinating everything and HAProxy w/ consul-template for dynamic routing. There were only two pieces that we had to write. The first (which we may open source, if we're given the time to clean it up and generalize it) is a small Go agent that runs on provisioned hosts, figures out its role based on instance meta data, pulls its configuration from Consul and handles deployment, both initial and subsequent when a new version is registered with Consul. The second piece is ensuring that CI generates Docker images as artifacts, pushes them to our private registry and updates Consul to indicate that there's new code to deploy.

It took us about a week to get this working and it's been mostly rock solid for almost a year now. Part of why it's been solid is that we understand exactly how every component of it works. The one problem we've had came from not understanding how HAProxy worked (never point HAProxy and an ELB...it will cache the NS resolution and ELBs can change IPs over time). If we'd tried something off-the-shelf, we'd have a much shallower understanding and, since it's not optimized for our use case, we would have run into many more issues than we've had. On the whole, I highly recommend rolling your own. The code that you will have to write is glue code that's really just replacing what would be configuration in something pre-built. I get that it seems imposing to people without devops experience, but between the tools that are available these days and articles like the one we're commenting about, it doesn't take a guru to get everything working seamlessly. Also, the tools from Hashicorp are fabulous. Use them whenever possible. No disclaimer necessary since I have no affiliation with them beyond using their tools and watching their talks on the subject.