Given how much effort you have to go through with etcd and discovery, why not use an MQ like RabbitMQ, that will take care of replication, etc as well?
First of all, because one of their explicit choices was to use HTTP. Using a persistent message bus like RabbitMQ has great advantages, but in terms of failure modes acts very differently to HTTP. For example, an upstream service failure would result in a timeout on getting a response (or ideally, a different instance of that service would respond to your message instead), rather than a more traditional TCP socket error or close.
Again, not necessarily worse, just different and unfamiliar.
The other reason is that this really only pushes the service discovery down a layer - how do you know where RabbitMQ is running? Of course, the same could be said for etcd - how do you know where etcd is running in order to query for more services.
Thanks for the explanation. I was looking for any dealbreakers with the MQ approach, but the concern you mentioned is valid. About pushing discovery down a layer, sure, but you'd do it the same way you'd query etcd, as you say.
RabbitMQ has decided to punt on any robust scheme for surviving network partitions. If you're building a distributed architecture and you care about availability and correctness you should just avoid it.
ekimekim summarized the reasoning quite well - primarily it was because I wed myself to HTTP early on in the project. To the question of knowing where the service registry itself lives, that is the one dependency which cannot (to my knowledge) be resolved dynamically. A persistent DNS record, which you update whenever the registry location changes, is the best I can think of atm.
Thank you for the response. You're right, I think discovery needs to be resolved statically, but a persistent DNS record hosting a server that can give you the rest of the info seems reasonable.
Again, not necessarily worse, just different and unfamiliar.
The other reason is that this really only pushes the service discovery down a layer - how do you know where RabbitMQ is running? Of course, the same could be said for etcd - how do you know where etcd is running in order to query for more services.