Ask HN: Who uses Kubernetes in production?

[the_gigi]

Kubernetes is making amazing progress. A lot of people are involved and even more people are watching closely. But, who actually walks the walk and runs production software in Kubernetes cluster?

[leetbulb]

I do! The hardware layer consists of physical machines running XenServer. Networking layer is 1Gbit WAN interfaces and 10Gbit LAN interfaces on "virtual switches" all wired together with pfSense. Gitlab-CI takes care of deploying just about all of the fabric on top of that including the images and app / system components / resources. Ingress is currently being overhauled, but right now it's essentially exposed HAProxy (pfSense) on top of dedicated, HA ingress VM's. Oh and it's all CoreOS. We're running three sites on it each with dev environments. Maybe a couple random API's, too. I haven't looked through all of the namespaces in a bit.

Edit: Gitlab-CI runners run on kubernetes as well using the dind images. Ingress nodes will soon be given public IP's. Public IP's are currently on CARP failover. After the gitlab-ci-multi-runner 1.1.1 release (allowing shared artifacts) and Kubernetes Deployment resources (providing a way easier deployment workflow and orchestration of pods), CI/CD is a breeze. We have dedicated nodes for MySQL (PXC) and ZooKeeper because these don't play well in the Kubernetes network environment - don't ask me to look at the examples ;) Currently running with Flannel for the overlay, but we're evaluating Calico and waiting on new Docker features to pull the trigger on something else... Multicast, isolated namespaces, VLAN's would be awesome :)

Edit2: I don't know why I keep saying "we" ... I've built and run this thing solo on top of programming... Not enough hours in a day...

[TheIronYuppie]

Can you drop me a note - Aronchick (at) google? We'd love to chat about your deployment :)

[leetbulb]

You got it!

[AkihiroSuda]

Kubernetes in Production in The New York Times newsroom https://kubeconeurope2016.sched.org/event/67f2/kubernetes-in...

[chews]

500 node clusters EVERY darn day. Glad they removed the 500 node the limit since kube 1.2 to allow clusters of larger size. We run primarily on google compute but we also run smaller clusters on Amazon.

[techthumb]

SoundCloud are moving to Kubernetes too: https://www.youtube.com/watch?v=5378N5iLb2Q

[hijinks]

We are running it in production

AWS with kubernetes 1.2.1 and calico as the overlay network. We have all our web apps in kubernetes and working on our background job apps next.

[TheIronYuppie]

We have tons of folks - any particular verticals I can highlight?

Disclosure: I work at Google on Kubernetes.

[aprdm]

Do you have examples of running it on premises without external internet access?

[nethier]

We're running many small (10s of nodes) clusters on metal with coreos. Networking is some in house stuff we've purpose built so we can get public ips on the pod.

Internal adoption seems to be going well so hopefully this grows.

[leetbulb]

Check out the new Ingress stuff for exposing services to public.

http://kubernetes.io/docs/user-guide/ingress/

https://github.com/kubernetes/contrib/tree/master/ingress/co...

The ingress controller runs on nodes with a public IP. When you create an ingress for one of your services, one of the controllers will pick that up and expose it. The ingress controller linked above is built on nginx and essentially just reconfigures nginx based on your ingress resource spec. Similar to when you define a LoadBalancer in a service with GCE, the service's external IP is set to the upstream ingress controller node's public IP.

[cweagans]

We use it at NBCUniversal for some internal things. Nothing public-facing, though. Not yet, anyway.

[yuvipanda]

We run it at Wikimedia for our Tool Labs setup, and are migrating more things to it over time.

[trungonnews]

Who's using google cloud container service?