[mikepurvis]

What are some of the limitations of managed k8s? For a lot of scenarios, it really seems like it could be an ideal set of tradeoffs in terms of who manages what and being able to still maintain some shred of platform agnosticism.

[tthayer]

The controlplane costs ~$45USD/year to run but so far my experience with it has been super positive. They support new releases quickly and if you're using managed node groups you can do blue/green version upgrades.

The main tradeoff for using EKS is the built-in CNI is limited by the number of virtual IPs you can assign to a given node type's network adapters. If you are running a bunch of small workloads you end up with very under-utilized compute and if you're using autoscaling that results in a lot of churn. We replaced it with weave-cni which has it's own set of headaches in regards to pod access to the control plane but it does allow us to run a basically unlimited number of small workloads on somewhat beefy instances. We set it up using spot pricing and the costs have been very low.

[mikepurvis]

Okay, thanks! My primary interest is for spot instance CI workers, so yeah, the deployment side is a non-issue and having a sane story for auto-scaling many small short-term workloads is exactly my scenario.