The per-packet processing overhead is a real and unresolved problem.
I'm using Rancher's networking now instead. It uses ipsec between hosts, so everything gets handled by code paths which have been optimized in the kernel, and performance is good (especially if you have a not-ancient CPU and have the AES-NI instruction - then wirespeed gigabit works with acceptable overhead).
I like Weave's decentralized architecture and wish it were realistic to use it.
If they are going to stick with a user-space solution they probably need to use DPDK or one of the other high-performance software defined networking toolkits, which tend to process packets using a SIMD approach.
If you check out the PDF that fons posted above, http://rp.delaat.net/2015-2016/p50/report.pdf, then you will see pretty extensive testing showing that Weave Net, flannel, and Docker Networking have similar VXLAN performance for unencrypted traffic. In all cases, it is good enough. Alas the testers were unable to get Calico working.
The question is: when do you want top performance for encrypted traffic? Most of users want encryption for the wide area or public cloud, and when they can't use a VPC. Our solution is pretty good for these cases. Obviously at some point we'll enable IPSEC too.
Widearea/public cloud & non-VPC use cases are my use cases.
I really wish this weren't true, but your solution is not pretty good yet. For now, if you need encryption, it's useless.
Machines spend their life handling packet overhead. Application performance suffers horrendously, and the scalability of the application goes from excellent to terrible.
Weave looks really good if you give it easy tests involving big packets. But if you give it a workload involving many small packets (which in today's microservices architectures is not exactly uncommon), it stops working.
IPSec is still pretty slow. While it might be fine for X DC stuff, without a real private network you're not going to find the low latency that makes microservices attractive.
The per-packet processing overhead is a real and unresolved problem.
I'm using Rancher's networking now instead. It uses ipsec between hosts, so everything gets handled by code paths which have been optimized in the kernel, and performance is good (especially if you have a not-ancient CPU and have the AES-NI instruction - then wirespeed gigabit works with acceptable overhead).
I like Weave's decentralized architecture and wish it were realistic to use it.
If they are going to stick with a user-space solution they probably need to use DPDK or one of the other high-performance software defined networking toolkits, which tend to process packets using a SIMD approach.