| It all about trade offs. We've built a CNI for k8s and have looked into all of the techniques described. It seems that Lyft's design is a direct reflection of their requirements. To the extent your requirement match theirs, this could be a good alternative. The most significant in my mind is that it's meant to be used in conjunction with Envoy. Envoy itself has its own set of design tradeoffs as well. For example, Lyft currently uses 'service-assigned EC2 instances'. Not hard to see how this starting point would influence the design. The Envoy/Istio model of proxy per pod also reflects this kind of workload partitioning. Obviously, a design for a small number of pods (each with their own proxy) per instance is going to be very different from one that needs to handle 100 pods (and their IPs), or more, per instance. Another is that k8s network policy can't be applied since the 'Kubernetes Services see connections from a node’s source IP instead of the Pod’s source IP'. But I don't think this CNI is intended to work with any other network policy API enforcement mechanism. Romana (the project I work on) and the other CNI providers that use iptables to enforce network policy rely on seeing the pod's source IP. Again, this might be fine if you're running Envoy. On the other hand, L3 filtering on the host might be important. Also, this design requires that 'CNI plugins communicate with AWS networking APIs to provision network resources for Pods'. This may or may not be something you want your instances to do. FWIW, Romana lets you build clusters larger than 50 nodes without an overlay or more 'exotic networking techniques' or 'massive' complexity. It does it via simple route aggregation, completely standard networking. |