[tobad357]

We have been using calico Cni with kubernetes with quite some success. It's a solution that assigns ips and then sets up bgp routing between your docker/kubernetes nodes. It makes it very easy to trace what's happening. Not as easy as a flat lan but still pretty easy.

Netstat -rn and you can see where the traffic is going or coming from. The added benefit is you can then bgp peer with other clusters and get routing across them

http://www.projectcalico.org

[alvelcom]

Calico looks good. As far as we understand calico works the same way as flannel/host-gw does, i.e. creates a route for all subnets in the cluster. Of course, calico is more advanced technology than flannel/host-gw, but the reason why we haven't tested it is following: they're different in control path but the same in data path. Our original intention was to test underlying linux-kernel mechanisms and understood how much we lose in terms of latency and throughput.

For small configurations flannel/host-gw is OK, I guess, but if one have several kubernetes clusters, flannel/host-gw becomes harder to maintain. That's a place where calico should be useful.

[tobad357]

Actually my understanding is a bit different. Flannel uses VxLan and acts as a overlay network (packet inside of packet) while Calico is a pure Layer 3 routing solution. This means some performance gains but less protocol support vs VxLan. I think it would be quite interesting to test