[zek]

I have this exact setup (vault/consul/nomad + glusterfs and zerotier for networking) and its pretty awesome. Still dependent on letsencrypt for SSL certificates. It runs plex amazingly well (though I am using some machines which are probably overkill for this purpose), even with the data coming from a glusterfs drive. Most of my nomad tasks can just launch anywhere because of gluster.

For internal DNS at least, you can just use consul. I set up dnsmasq to forward to consul on all of my machines which is super convenient (esp when that DNS just points to a docker container ipv6 address on the zerotier network, not port remapping on networking insanity needed)

[ckrailo]

Any additional build details from you or Legogris?

Haven't considered using glusterfs but the rest of those pieces, absolutely.

Only piece I could find similar to the topic is https://www.mockingbirdconsulting.co.uk/blog/2019-01-05-hash....

[Legogris]

After a quick scroll-through I have a very similar setup. One thing to watch out for is that you really want to set up TLS and ACLs for Consul, Vault and Nomad as early as possible (maybe ACLs for nomad is not as critical to have in place from day one) - if you can avoid bootstrapping that on an already running cluster you save yourself a lot of head-scratching.

Integrating these three systems, with Terraform on top, is pretty time-consuming with all the policies and TLS certificates, but it seems pretty smooth to maintain after the initial setup.

[ckrailo]

Nice, thank you for the TLS/ACL hint!

I'm learning terraform/consul/nomad/vault as I build this thing, so I think the learning curve for me is extra painful compared to some of y'all.

The docs seem really good if everything is cloud-hosted and really nonexistent if you're running it all on your own metal.

I'm thinking I'll build out my test stack with vagrant. I can use Dropbox or Keybase or something similar to store and sync my terraform state if I need to. Any other hints or helpful readings?

[Legogris]

Cheers, haven't checked out zerotier before. I'm curious, do you run GlusterFS on nomad or standalone? Seems like it could work fine as raw_exec, but I'm not sure if that's a good idea or not.

[zek]

kinda a late response (sorry) but I run it standalone, setup via ansible. I have ansible scripts that bootstrap everything "below" nomad so it sets up zerotier, then vault, then consul, then dnsmasq (pointing to consul) and then nomad. You could probably run gluster in nomad but given that I give most nomad tasks a gluster directory that feels odd.

The only thing I still need to figure out about this setup is that I currently use a single glusterfs volume for most of my nomad tasks, I would love to have a nomad integration that could provision and mount the gluster volume when I specify a volume for a docker task in nomad.