[eropple]

Kubernetes requires machines big enough to run all your containers. Those machines are the bins. Your containers are the packages. Fitting your containers in such that there is no criticality overlap (in AWS, that all instances of service X are spread across machines in different AZs) and that there is room for immediate scaling/emergency fault recovery (headroom on the machines running your containers) gets expensive. You're buying big and running little, and that comes with costs.

Meanwhile, in AWS, you already have pre-sized blobs of RAM and compute. They're called EC2 instances. And then AWS pays the cost of the extra inventory, not you. (To forestall the usual, "overhead" of a Linux OS is like fifty megs these days, it's not something I'd worry about--most of the folks I know who have gone down the container-fleet road have bins that are consistently around 20% empty, and that does add up.)

You may be the one percent of companies for whom immediate rollout, rather than 200-second rollout, is important, and for those companies a solution like Kubernetes or Mesos can make a lot of sense. Most aren't, and I think that they would be better served, in most cases, with a CloudFormation template, an autoscaling group with a cloud-init script to launch one container (if not chef-zero or whatever, that's my go-to but I'm also a devops guy by trade), and a Route 53 record.

You're basically paying overhead for the privilege of `kubectl` that, personally, I don't think is really that useful in a cloud environment. (I think it makes a lot of sense on-prem, where you've already bought the hardware and the alternatives are something like vSphere or the ongoing tire fire that is OpenStack.)

[andrewstuart2]

I know you're answering the question of bin-packing, but after two years of experience with it, I can say that for me, bin-packing is one of the smallest benefits (though it sells very well with management), though perhaps a baseline requirement these days. The real benefits, in my experience, stem from the declarative nature of cluster management, and the automation of doing the sensible thing to enact changes to that declarative desired state.

[eropple]

Sure. CloudFormation exists for that, though, and both its difficulty and its complexity are way overstated while also letting you manage AWS resources on top of that.

And it doesn't cost anything to use.

[andrewstuart2]

Eh, there are a lot of terrible things I'd rather put myself through than writing another CloudFormation template for any sort of complex infrastructure. It could have been made easier and more readable if my company had allowed the use of something like Monsanto's generator [1], but creating ASTs in JSON is not my idea of a good user experience.

[1] https://github.com/MonsantoCo/cloudformation-template-genera...

[eropple]

I maintain auster[1] and am a contributor to cfer[2] for exactly that purpose. ;) CloudFormation really isn't a rough time anymore, IMO.

[1] - https://github.com/eropple/auster

[2] - https://github.com/seanedwards/cfer

[dvlsg]

If you know those tools exist, maybe. I just put together a new project using cloudformation (technically serverless, but it turned into 90 percent cloudformation syntax anyways), and it was pretty rough.

[eropple]

Maybe it's just me, but as a programmer the first thing I ever asked when looking at the wall of CloudFormation JSON was "so how do we make this not suck?".

Our job is not just to automate servers, it's to automate processes, including stupid developer-facing ones.

[wahnfrieden]

If your only experience with CloudFormation is hand-written JSON, it's worth another look.

We used to use troposphere, a Python library for generating CloudFormation templates, but have since switched back to vanilla CloudFormation templates now that they added support for YAML. We're finding it's much nicer to read and write in plain YAML. We're also now using Sceptre for some advanced use cases (templatizing the templates, and fancier deployment automation).

[sjellis]

> If your only experience with CloudFormation is hand-written JSON, it's worth another look.

Strongly agree.

YAML and sensible formatting conventions really do transform the usability of CloudFormation.

[owaislone]

And so does terraform which is pretty awesome!

[eropple]

Terraform requires significant infrastructure to get the same state management and implicit on-device access that CloudFormation's metadata service does. A common pattern in systems I oversee or consult on is to use CloudFormation's metadata service (which is not the EC2 metadata service, to be clear) to feed Ansible facts or chef-zero attributes in order to have bootstrapped systems that do not rely upon having a Tower or Chef Server in my environment.

The Terraform domain spec is not sufficiently expressive (just look at the circumlocutions you need to not create something in one environment versus another). It's way too hard to build large modules, but the lack of decent scoping makes assembling many small modules difficult too. Worse, the domain spec is also requires HCL, which is awful, or JSON, which is a return to the same problem that cfer solves for CloudFormation. One of my first attempts at a nontrivial open-source project was Terraframe[1], a Ruby DSL for Terraform; I abandoned it out of frustration when it became evident that Terraform's JSON parsing was untested, broken, and unusable in practice. Out of that frustration grew my own early CloudFormation prototypes, which my friend Sean did better with cfer.

If you're looking for an alternative to CloudFormation, I generally recommend BOSH[2], as it solves problems without introducing new ones. Saying the same for Terraform is a stretch.

[1] - https://github.com/eropple/terraframe

[2] - https://github.com/cloudfoundry/bosh

[Rapzid]

Cloudformation is not without its problems, even still. I feel you overstate Terraform's issues, though there is tons of valid criticism to go around. I would say Terraform really shines in areas CF still does not.

We use Terraform for our foundation and a clever Cloudformation custom resource hack to export values out to use in Cloudformation stacks where appropriate(use with Serverless services, etc). Works great for us; Terraform has seen significant(surprising even if you haven't looked at it in 6+ months) development over the past year.

[ttul]

Immutability, in other words.