[vdloo]

This is very cool. I'm wondering if decentralized overlay 'mesh' networks will become more prevalent in the future. Overlays have obvious benefits for multi-cloud setups if latency isn't an issue for the services you run through the network. I can imagine this becoming a more popular technique in the future as cloud instances become cheaper and more people are willing to make the performance trade-off for convenience. Additionally it can be a great pattern to fight vendor lock-in.

I run a similar concoction for my 'home network' which consists of various mobile devices like my Android phone and some cloud servers, but instead of ZeroTier and I use CJDNS in combination with Consul. The code for it is on github in the vdloo/raptiformica repo. I ran into various issues with the difference in latency between nodes. I think most people run Consul in a very homogeneous environment (like in one datacenter), but maybe perhaps the differences between using it cross-cloud is not enough to cause problems. I'm wondering if there were some Consul settings that the author had to tweak (and how) for stability and if there were any unexpected issues.

One thing that caused me problems with Consul on overlay networks was an ARP cache overflow. DigitalOcean also ran into that running Consul at scale in their DC if I recall correctly: http://youtu.be/LUgE-sM5L4A I noticed that if I put enough Dockers on one host (like 50 - 100) in an overlay network and tried to run Consul on top of that things would start to overflow, presumably because of all the network abstraction and redundancy. I'm wondering how many machines the author had in one Consul cluster and if they tested to what amount of nodes this setup could scale.