On the topic of "why do we need a distributed KV store for an overlay network?" from the blog: there's a good blog post about why Kubernetes doesn't use Docker's libnetwork.
Thanks! I had almost forgotten about the container networking mayhem. Would love to find out what progress has been made over the past six months since the blog post was written.