[benlivengood]

I've started building a kubernetes cluster (Talos Linux) across town with wireguard between various houses. ZFS boxes for persistent volumes (democratic-csi) in each "zone" with cross-site snapshot replication and Gateway (Traefik) running at each site behind the ISP. CrunchyPGO allows separate StorageClasses to easily split the leader/followers up.

[nickorlow]

Have had issues w/ doing k8s over residential wan once I had enough hosts in my cluster

(though they were halfway across the US from each other, and not town)

[benlivengood]

So far everything is under 15ms apart, but it is a small number of nodes so far. Did you mostly have trouble with etcd?

[nickorlow]

Yeah, etcd was the main culprit, but latency was 150-300ms in my case. At 3 nodes, it was relatively stable (had an issue every week or so that lasted < 5 min), but at 4 the camel's back broke.