[PascalW]

I'm doing something similar though I've opted specifically to _do_ use Kubernetes via k0s [0]. It works wonderfully well and allows me to use most things that are available in the k8s ecosystem like cert-manager [1] and external-dns [2]. All configuration is stored in Git and the server itself is basically disposable as it only runs k0s and some very basic stuff like iptables rules.

I see this sentiment quite a lot that k8s is too complex for small scale setups, but in my experience it scales down very well to small setups. Granted, you do have to know how to work with k8s, but once you learn that you can apply that knowledge to many different kinds of setups like k0s, bare metal, EKS/AKS etc.

[0] https://k0sproject.io/

[1] https://cert-manager.io/

[2] https://github.com/kubernetes-sigs/external-dns

[znpy]

I'm playing with k0s and it seems it doesn't play nice with firewalld.

With firewalld active containers cannot do networking, not even with hosts jn the same lan.

Everything else works beautifully though.

[PascalW]

I'm using iptables myself and it works fine, though you have to make sure that traffic on the kube-bridge interface is allowed.

With iptables:

  -A INPUT -i kube-bridge -j ACCEPT
  -A OUTPUT -o kube-bridge -j ACCEPT

Other than that I've configured iptables to drop all incoming traffic except a few whitelisted ports.

[kakoni]

Do you have any estimates how resource hungry k0s is? Ran few resource constrained k3s clusters, where 25% of cpu was always spent on running k3s itself.

[PascalW]

Very similar. I guess it's really k8s (the control plane) itself that is so resource intensive. Looking with top right now kube-apiserver, kubelet, kube-controller, kine and k0s use 13.5, 12.5, 5.6, and 3.0 % CPU respectively. Obviously it fluctuates quite a bit, but seems to be around 25-30% of 1 CPU core too. Also uses about 500-600mb of memory.

So yes, it definitely takes quite a bit of resources. I'm running this on 4 CPU cores and 6 GB memory, so 25% of 1 core and some 600mb of memory still leaves plenty of resources for the services. On a more philosophical note (as was mentioned below in this thread), it is a bit wasteful perhaps.

[Already__Taken]

Just done a very quick search but it seems like k3s is the better choice so what did you like about k0s?

[PascalW]

Curious what makes you think k3s is the better choice? The only reason I ended up going with k0s was that I had problems getting k3s working well behind a locked down firewall. With k0s that was pretty easy.