[3np]

It depends on your size. For a fairly minimal close-to-best-practices you'll need for each DC, each on a separate physical host (I may be missing something):

  3 x Consul server
  3 x Nomad server
  2/3 x Vault server

It's long since I operated k8s but IIRC I think you can get similar capabilities and redundancy with 3-5 machines?

That's before you start looking at actual runner nodes, load balancers, proxies, logging and monitoring infra, etc...

Unless you cheat (which I think many do) or you're big enough, that overhead can be meaningful.

[schmichael]

(Disclosure: Nomad team lead)

FWIW we recognized this was too much overhead for many users. Nomad 1.3 supports service discovery so you can start without Consul, and 1.4 will support secure variables to get folks farther along without requiring Vault.

So 3 Nomad servers should give you a pretty featureful and highly available cluster these days.

[orthecreedence]

Yeah or, like, spin up three medium servers in different zones and have each server run all three services. We did that for a production setup for years and it worked fantastically. There's no need to have nomad/consul/vault all on different servers unless they are significantly underpowered or the workloads are crazy.

If best practices say otherwise, then maybe they should be reconsidered.

[3np]

Sure, but at this point there's so much else we get from Consul that, like, what's the point...

I guess the path is set but I'd personally much prefer having a recognized deployment scenario be hosting Consul server and Nomad server on the same physical machines, and accommodating (be it through code or just docs) for making that play well with security, certs, and resource usage without becoming a confounding mess.

Even Vault, if the operator accepts and/or mitigates the sidechannel aspects - from a security perspective that still shouldn't be a step down from anything Nomad-specific?

Seeing as HC already provides solutions for all of these supposed to be serving for Nomad, doesn't it make more sense to make them play together smoother and nice on the same machine rather than reinventing a lesser wheel for each of them?

[nicolas_t]

Entirely true, but I also think that neither k8 nor Nomad are that useful if you're not at a scale where the above is negligeable? It costs roughly 500 usd a month on aws for those 9 servers.