[zerogvt]

It seems that k8s has won the deployment race by and large. I see a lot of success stories around (I'm hearing nice things from the DevOps teams in my organization as well). Yet I'm curious to hear a few cases where things did not pan out quite right.

Note: The 5-15s DNS problem seems a pretty serious one. Weird that it didn't get more publicity (and a proper fix).

[zimbatm]

There are a lot of things that can go wrong with K8s but there is always a way to fix them. For example a common mistake it to forget to allocate limits on pods, which then brings the worker node to capacity. I think the failure scenario is soft, it's just going to cost more engineering time to figure out how to upgrade the cluster to the new version, find out why this network overlay isn't performing as expected or debug this external resource that isn't being allocated properly, configure RBAC properly, play with various resource deployment strategies, tune how pods are being moved during a node auto-scaling event... The nice thing is that at the end it gives a unified API for all of the things, it forces some consistency in the infrastructure.

My personal rule of thumb is that unless the client specifically need auto-scaling or have more than 100 services to run, have a 5 people devops team, just use Terraform.

For a small number of servers a better strategy is to have a base image with Docker and monitoring, and use Terraform to deploy the infrastructure. CI can then use docker-compose to deploy the containers onto the hosts directly. This approach is much more stable and doesn't require to learn as many things as K8s. This can be run by a 1 man DevOps team without a sweat.

[geggam]

Hard to fix something broken by design.

Using TCP as the main message bus then using layers upon layers of NAT needs to be revisited, routing is the solution.