[tso]

> Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift.

The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding. Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

: The control points that k8s exposes are not simple "drop in your provider here" bits of integration. You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them, and more access control to implement and verify.

[theptip]

My personal experience (4 years on GKE in production) has been the opposite; running on k8s has abstracted away a number of things that I’d otherwise have to engineer.

Volumes just get attached (using PersistentVolumeClaims), and automatically migrate to a new node of the original pod dies. Vs. having to do some sort of rsync between nodes to keep disks in sync.

Secrets get encrypted by k8s and mounted where needed. I would agree that RBAC is a bit tricky but I don’t think it’s harder than IAM provisioned with Terraform.

If you are not using a service mesh for your VMs then you don’t need one in k8s. (I don’t use one, and rolled TLS to the pod in less effort than it would take to maintain TLS to the VM). The reason you want a service mesh is to abstract TLS and retry mechanics from the application layer - i.e. make your service authors more productive. If you don’t use a service mesh then you are back to managing TLS per-service, which is where you are with VMs already.

There are definitely more services you _could_ run, but in my experience these are additive, I.e. they are extra work, but give you a productivity boost.

Anyway, YMMV and I haven’t operated a system as large as Coinbase, so I could be missing something. Interested in hearing others’ experiences though.

[rumanator]

> As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding.

The problem with that assertion is that it does not make any sense at all. For instance, storage and config/secrets is already supported out-of-the-box with Kubernetes. Even so, complaining about storage with Kubernetes is like complaining about EBS or EFs or arguably S3 in AWS. And if you feel strongly about service meshes then you really aren't forced to use them.

> Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

There really isn't. For example, if they go with managed Kubernetes solutions then the only thing they need to worry about is to actually design their deployments, which would be very strange if they couldn't pull off. That's a ramp-up project for an intern if the solution architecture is already in place.

> You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them

There really aren't. Kubernetes' equivalent to EBS is either setting up a volume or a persistent volume claim on a persistent volume. Just state how much memory you want and you're set.