[ungawatkt]

I've been trying out Nomad on a home server setup, its still not as simple as Swarm and has some hard edges to learn around, but overall it hasn't been a bad experience, definitely easier than my attempts at k8s. And Nomad got basic built in service discovery in the last couple versions, which was the main thing turning me away (it used to require running Consul).

[maxnoe]

Can you describe your setup?

I tried setting up nomad but couldn't figure out service discovery, external ingress / letsencrypt in the time I allotted myself...

[ungawatkt]

Nomad running as a server (manager) and client (worker) on the main server, and as a client on a rpi3, through systemd on both, the install [0] and deploy [1] instructions worked well enough. It has constraints to control which client takes what workload (ex. I don't care which runs ddclient, but photoprism can't move to the rpi, and the sensor readers can't leave the rpi).

Caddy for reverse proxy and tls handling running as a service, so all ingress goes through that, there wasn't too much exciting there. I was porting over a docker-compose workload, so most services have a static port and I just route to the port in the caddyfile (I passed my router as the DNS for the reverse proxy, so its by hostname). I'll get caddy to use nomad for service addresses at some point.

But for photoprism, I've got the service discovery set up to its database, it's kind of awkward since the nomad native discovery goes into env vars or a file through the `template` block [2], but it does work.

Nomad is pickier about setting resource limits, so I had to actually set those to something reasonable.

I've mostly got Docker based services, so this reference [3] has been useful. There's a couple ways to mount the volumes, which is annoying, and there's some gotchas around docker image handling (short shorty is don't use the `latest` tag).

I probably haven't organized my jobs/groups/tasks well (the analog to k8s pods hierarchy), but that's for later.

[0] https://developer.hashicorp.com/nomad/tutorials/get-started/... [1] https://developer.hashicorp.com/nomad/tutorials/enterprise/p... [2] https://www.hashicorp.com/blog/nomad-service-discovery [3] https://developer.hashicorp.com/nomad/docs/drivers/docker

[e12e]

> services have a static port and I just route to the port in the caddyfile

Just so I understand - this means deploying a new service involves building a docker image, deploy it, then manually update the caddyfile (ie manual ingress)?

> There's a couple ways to mount the volumes, which is annoying,

Fwiw this is annoying on multi-node docker swarm too - i even consider proper volume support to be one of the strongest arguments for considering k8s even for somewhat simple setups.

[ungawatkt]

yeah, that's what I have now. I see the path to letting nomad deal with the addressing (rather than specifying static ports), but setting up a route in caddy for a new service would still be manual. I expect you could do some fancy scripting/go templating with the caddyfile and template block to make it spin up new routes more "automatically", but at that point I think you'd be better off seeing what Consul could do for you. Or traefik discovery with tags, I expect that could be convinced to work with Nomad, not that I've tried. (edit, seems like yes [0])

And I'd agree that the volume story isn't great compared to what I know of k8s. There's info out there for setting up NFS volumes, or something like portworx or ceph, but that's going beyond what I want to do for 1-3 pet nodes on a home server, I can deal with volumes staying bound to a node.

[0] https://traefik.io/blog/traefik-proxy-fully-integrates-with-...

[e12e]

Been using traefik with docker swarm for ingress - and it works. But doesn't exactly feel slick imnho :/

[nesarkvechnep]

You won't find a resource which encompasses that. I have the feeling its purposeful so you pay for a course or certificate.

[regularjack]

Are you saying course authors are removing other people's content from the Web?