[chris_marino]

Great post!

The results of these benchmarks do not surprise me at all. To me, they all fall in to the category of 'less (overhead) is more (performance)'. With VXLAN encap being the obvious example of greatest overhead.

I think its also worth mentioning that k8s networking is being enhanced in v1.2 to support isolation and multi-tenancy through ThirdParty resources (back end network solutions). The alternatives included in the benchmarks aren't going to be able to support these kinds of network policy as is.

And, unfortunately, things get a more complicated when you want to provide more than simple reachability (which is all that k8s asks for today). The tradeoff is to be able to control the packets with the lowest overhead possible. VXLANs will give you isolation, but at the cost of encapsulation. Stacking bridges and tunnels and distributing VNIDs/routes not only introduces more latency, but becomes another multi-host coordination problem (matching tunnel IDs, etc).

We're working on a new way to build cloud native networks that avoids the encap, but still lets you control all the packets.

You can learn more at http://romana.io if you're interested.