[scripni]

This runs on AWS managed EKS these days, this talk goes into more detail about Netflix's special sauce around the k8s control plane: https://www.youtube.com/watch?v=vaTOiXR2KSM

Netflix actually has much fewer cells than you'd expect btw, their special sauce IMO is federation and using a small subset of k8s APIs.

[__turbobrew__]

I am surprised a company at that scale is running on managed EKS, maybe I underestimate how large the clusters are.

[zbentley]

EKS can get pretty damn big, well into the thousands of nodes without much special tuning, and beyond that with some care and control plane monitoring. Expensive, though.

[__turbobrew__]

> Expensive, though.

That is my point. I work at a large multinational and we run tens of thousands of kubernetes nodes on-prem and Im pretty sure that would be in the hundreds of millions of dollars per year to run in EKS. We run on-prem nodes about equivalent to c6a.32xlarge and even with 2 year reserved pricing you are looking at $17k/year/node. At 20000 nodes you are looking at $340 million/year, not including egress fees or any other AWS service charges (such as EBS).

I can tell you with certainty that the all-in costs to run kubernetes on-prem (including staffing costs) is a lot less than $340 million/year AND we don’t have vendor lock in. In total we have 7 full time engineers building and running on-prem kubernetes. The more nodes you have, the more it makes sense as the team size is mostly independent of the number of nodes, so that team of 7 could also run 40000 nodes without issues. The cost becomes dominated by the capex to purchase hardware. I would say team size is log(nodes).

For a company the scale of Netflix, I would assume the math is similar — especially since they already have in house expertise to run their own hardware — but maybe they get a very steep discount from AWS.