[iasay]

It’s really complicated when something goes wrong. That is my only criticism. Particularly in the various CNI layers out there. You really have to know exactly how everything works to get through those days and that is beyond the average person who can create a docker container and push it into the cluster which is the usual success metric.

[p_l]

Networking is complex, unfortunately, and cloud networking has a legacy of trying to support things that never should be supported (stretched L2s, 10./8 everywhere, etc.)

Things get much simpler if you try to limit CNI complexity by going towards at least conceptually simpler tooling that matches original, pre-CNI design of k8s, IMHO.

[jpgvm]

99% of people aren't going to use a different CNI plugin to what their managed distribution ships with. Same goes for peeking under the covers of storage plugins, kubelet config, etc.

You pay AWS/GCP for that these days and just use the API.

[iasay]

It was AWS who had trouble fixing our CNI issues…

[nouveaux]

If AWS broke your CNI that caused you problems, it is also possible for AWS to break networking.

I have used Managed K8s for 4 years and literally never had any problems with CNI. My clusters runs with no problems.

[iasay]

Yes we have no problems for 2 years. Then we had problems.

[Noughmad]

In what cases would I have to worry about that?

[nprateem]

If it breaks. If you have an outage because of weird K8s networking issues (and I've seen them) you'll suddenly care very much.

[iasay]

Yes same. We had one which we had to recreate a whole EKS cluster because AWS couldn’t fix it too.

I don’t sleep these days.