Nomad running as a server (manager) and client (worker) on the main server, and as a client on a rpi3, through systemd on both, the install [0] and deploy [1] instructions worked well enough. It has constraints to control which client takes what workload (ex. I don't care which runs ddclient, but photoprism can't move to the rpi, and the sensor readers can't leave the rpi).
Caddy for reverse proxy and tls handling running as a service, so all ingress goes through that, there wasn't too much exciting there. I was porting over a docker-compose workload, so most services have a static port and I just route to the port in the caddyfile (I passed my router as the DNS for the reverse proxy, so its by hostname). I'll get caddy to use nomad for service addresses at some point.
But for photoprism, I've got the service discovery set up to its database, it's kind of awkward since the nomad native discovery goes into env vars or a file through the `template` block [2], but it does work.
Nomad is pickier about setting resource limits, so I had to actually set those to something reasonable.
I've mostly got Docker based services, so this reference [3] has been useful. There's a couple ways to mount the volumes, which is annoying, and there's some gotchas around docker image handling (short shorty is don't use the `latest` tag).
I probably haven't organized my jobs/groups/tasks well (the analog to k8s pods hierarchy), but that's for later.
> services have a static port and I just route to the port in the caddyfile
Just so I understand - this means deploying a new service involves building a docker image, deploy it, then manually update the caddyfile (ie manual ingress)?
> There's a couple ways to mount the volumes, which is annoying,
Fwiw this is annoying on multi-node docker swarm too - i even consider proper volume support to be one of the strongest arguments for considering k8s even for somewhat simple setups.
yeah, that's what I have now. I see the path to letting nomad deal with the addressing (rather than specifying static ports), but setting up a route in caddy for a new service would still be manual. I expect you could do some fancy scripting/go templating with the caddyfile and template block to make it spin up new routes more "automatically", but at that point I think you'd be better off seeing what Consul could do for you. Or traefik discovery with tags, I expect that could be convinced to work with Nomad, not that I've tried. (edit, seems like yes [0])
And I'd agree that the volume story isn't great compared to what I know of k8s. There's info out there for setting up NFS volumes, or something like portworx or ceph, but that's going beyond what I want to do for 1-3 pet nodes on a home server, I can deal with volumes staying bound to a node.
Caddy for reverse proxy and tls handling running as a service, so all ingress goes through that, there wasn't too much exciting there. I was porting over a docker-compose workload, so most services have a static port and I just route to the port in the caddyfile (I passed my router as the DNS for the reverse proxy, so its by hostname). I'll get caddy to use nomad for service addresses at some point.
But for photoprism, I've got the service discovery set up to its database, it's kind of awkward since the nomad native discovery goes into env vars or a file through the `template` block [2], but it does work.
Nomad is pickier about setting resource limits, so I had to actually set those to something reasonable.
I've mostly got Docker based services, so this reference [3] has been useful. There's a couple ways to mount the volumes, which is annoying, and there's some gotchas around docker image handling (short shorty is don't use the `latest` tag).
I probably haven't organized my jobs/groups/tasks well (the analog to k8s pods hierarchy), but that's for later.
[0] https://developer.hashicorp.com/nomad/tutorials/get-started/... [1] https://developer.hashicorp.com/nomad/tutorials/enterprise/p... [2] https://www.hashicorp.com/blog/nomad-service-discovery [3] https://developer.hashicorp.com/nomad/docs/drivers/docker