[yebyen]

You can afford ~5 preemptible instances for the cost of one full-time instance. I think you are severely overestimating the harm that is done by preempting/terminating instances in a prod Kubernetes cluster that is capably configured for full HA and distributed across zones. That's what it's made for – nodes are as disposable as pods. They may come and go as they please, and they'll be replaced as needed by the self-healing nature of the cluster and the scaling group.

(Why save a few dollars when you can get extra capacity instead? Especially when you have $100k of someone else's money to burn... so, for real though, anyone who has actually tried this configuration can chime in and confirm that it works as well as I imagine.)

I'm pretty sure that periodically shutting down some nodes is a boon for cluster utilization, too. One of the things that Kubernetes does not do on its own, is load rebalancing. You can configure the autoscaler to recognize when nodes are overprovisioned, and let it drain a few and shut them down... or you can let the preemptible nature of (some/all of your) nodes do it for you.

(Why not both? Some nodes are getting killed, or you're paying for resources you don't use, so... one way or another.)

[user5994461]

Your message obviously implies that you have no experience with what you are suggesting, hence why you don't realize the downsides.

[yebyen]

"No experience" is a stretch. I'll concede, no significant Prod experience to speak of.

I got introduced to the Kubernetes ecosystem when Deis Workflow was rebuilt for it. Today, I'm a core contributor on the team that is building the fork of that project.

(It is a side thing for me. I'm very interested but I'm not spending hundreds on infrastructure every month. I have an infrastructure team at $dayJob, and they are not doing K8S at all, if you are having a hard time understanding how exactly I got here.)