[chrissnell]

We've seen similar savings at our company. We have deployed Kube on a 6-node cluster of CoreOS nodes with 512GB each. These are dedicated servers hosted at Rackspace. We're about 30-40% utilized on RAM and maybe 15-20% on CPU. To host a similar set of services on our older Openstack environment would require at least 2-3x the number of servers. The cost savings isn't even the best part. Kubernetes has allowed us to build a completely self-service pipeline for our devs and has taken the ops team out of day-to-day app management. The nodes update themselves with the latest OS and Kube shifts the workload around as they do. This infrastructure is faster, more nimble, more cost-effective and so much easier to run.

This is the best infrastructure I've ever used in twenty years of doing ops and leading ops teams.

[thockingoog]

You folks don't know how much it means to us to hear that people are finding success with Kubernetes. Thanks for using it. We'll try to keep pushing the envelope.

[chrissnell]

Thanks, Tim. Y'all have been awesome. Thanks for the quick response to GH issues and Slack questions. I hope that we can speak at a conference someday and tell the world about how much more fun and easy Kubernetes made our jobs.

[brokebook]

so true, the self service aspect is indeed amazing. once developers or qa people grok the concepts and api it can do wonders to your productivity.

also, working with k8s will probably spoil you, it's pretty annoying to "go back" to other environments, where you're confronted with problems which would be effortlessly solvable in kubernetes.

[chrissnell]

It completely spoils you. My team got to do an Openstack cluster migration this month and there was definitely some grousing. The workflow of traditional private clouds is just so tedious and flaky. We could grow our Kubernetes cluster 10x without hiring any additional engineers for the ops team.

[TheIronYuppie]

I can't agree more with others in the thread. This entire comment warms the cockles of my cold dead heart.

I know Silicon Valley folks are infinitely pessimistic and/or grandiose, but this is LITERALLY the reason I got into this job.

Disclosure: I work at Google on Kubernetes

[aedocw]

Have you compared the cost of maintaining your own CoreOS infrastructure at RAX just for kubernetes to using Googles Container Engine? If your services are all containerized and deployed via k8s to begin with, seems like you wouldn't have much reason to maintain your own infra at that point.

[chrissnell]

Yes, absolutely. We ran a four month experiment on GCE--we built an off-site logging cluster fluentd+Elasticsearch+Kibana. The performance was decent but the cost of RAM and disk are way higher.

I will tell you that the economics are most definitely not there. This is a common misconception amongst the HN crowd in general--that public cloud infra is cheaper. For small footprints, public cloud makes sense but once you get into the larger footprints (300+ instances), it's far cheaper to lease dedicated hardware or DIY in colocation. We're running on approximately 40 dedicated rackmount servers for Openstack and 6 for Kubernetes. To get the equivalent amount of disk and RAM, we would pay 2-3x at AWS or GCE. We could probably cut our cost by an additional 30% by moving what we have to colo but we would lose some flexibility and would have to take on additional headcount.

From a maintainability standpoint, GCE makes Kubernetes easy which is a good thing if you've never run it before. It's not that hard to run it yourself, though. A senior-level systems engineer will be a Kube master after about two months of use. Just guessing, I think it takes about 1/4 of an engineer-week to support our Kube cluster for a week. I think we could grow our cluster 20x without a significant workload increase for our ops team.

We are in the process of automating the last few manual aspects of our Kubernetes infra: load balancing and monitoring. We're building these in the style that we've built the rest of our pipeline: YAML files in a project repo. Simply drop your Datadog-style monitoring/metrics config and your load balancer spec in your project's Github repo and the deployment pipeline will build out your monitoring, metrics, and LB automatically for you.

[illumin8]

I'm curious - did you model reserved instance pricing, or on-demand pricing in your comparison? AWS typically charges ~1/3rd the price for a 3-year reserved instance vs. on demand pricing. This comparison would be much more apples to apples if you are purchasing hardware with CapEx that typically depreciates over 3-5 years.

[jturolla]

This is really interesting. Are you planning to open source this pipeline automation? I'd be interested.