[nemo136]

> 50 machines at hetzner

- install machines with ansible (using hetzner scripts for OS install)

- machines communicate over vswitch/vlans, external interfaces disabled whenever possible. Pay attention to the custom mtu trick.

- harden machines, unattended-upgrades mandatory on each machine

- ssh open with IP whitelists from iptables on gateways

- machines organized as k8s clusters, took ~1 year to have everything working cleanly

- everything deployed as k8s resources (kustomize, fluxcd, gitops)

- use keepalived for external IPs with floating IPs for ingress on 3 machines per cluster

Machines are managed as cattle, it takes <1h+ hetzner provisioning time to add as many machines as we need.

[Volundr]

> Pay attention to the custom mtu trick.

I wish Hetzner made this more clear up front. Maybe a big red banner on the vSwitch page. I can't count the number of hours I've spent troubleshooting network issues at Hetzner that came down to MTU.

[johne20]

As someone who is trialing Hetzner is there a link to info on this?

[Volundr]

The vSwitch docs tell you to limit MTU to 1400 https://docs.hetzner.com/robot/dedicated-server/network/vswi.... If you fail to do this it'll default to 1500 I think and manifest in unpredictable ways. Like being able to fetch updates one moment, then being unable to connect to the update server a few minutes later.