[MrSaints]

I'm quite curious about your scale / size of your team.

Most of the moving parts for k8s are easily handled if you go for a managed service.

Using Nomad alone is fine, but with Consul in the mix, it requires quite a bit of set-up (especially for some degree of HA), and from my experience, is no less harder than using something like k3s.

Overall from my perspective, it just seems like Nomad + Consul sits in two places.

Either for an org. small enough where HA is not a concern, so setting it up, and running it "on-premise" is trivial. Or, for an org. large enough where you can have a team dedicated to setting it up, and managing it to ensure it meets various SLAs.

Genuinely curious to know what's your experience been like, and if it matches up with this.

[kawsper]

We're 1.5 people to do ops works, but no one is full time on it, I'm the main responsible and I have someone that helps me when he's interested. We are 5 developers.

Our whole platform is between 10-50 EC2 machines running a Nomad cluster, Nomad manages our Docker containers and with services backed by RDS.

I think managed services were in their infancy when we did our initial research back in 2017/2018, Tectonic+Kubernetes with CoreOS looked promising but they were bought by Red Hat and probably rebranded/merged/disappeared into OpenShift. EKS was in beta an only available in the US (we're in EU).

We did try Rancher but we hit issues with it.

I don't know if K3S existed yet, but just looking at the diagram on their website it does look quite interesting.

We launched Consul first and started defining all of our services, and after that we started moving applications into Nomad.

HA has been quite easy with Terraform on EC2. We build "golden-images" with Packer, and then launch them with Terraform, upgrading Consul is adding 3 new servers, making sure things are stable, and then removing the 3 old servers.

[mrweasel]

> Most of the moving parts for k8s are easily handled if you go for a managed service.

Which large parts of Europe currently cannot do, or are at least unwilling to risk doing, due to Schrems II. Amazon, Google and Azure are the three providers with the best managed Kubernetes services, or that least those who require least involvement from the user.

Otherwise I completely agree, Kubernetes is less of an overhead, if you can rely on managed services. I do question the idea that Nomad can't easily do HA. If you can build an HA environment for Kubernetes, then you can just as easily do the same for Nomad.