[zumachase]

Because we needed geographic distribution so that we don't end up hairpinning our users, and they only run a single service so the value prop is much lower. We use route 53 to do geodns across a number of cheap instances around the world (which is also nice, let's you pick regions with cheap bandwidth but good latency to major metro areas). We currently have TURN relays in Las Vegas, New York, and Amsterdam and that gives us pretty good coverage (sorry Asia...you're just so damn expensive!).

But all of our APIs sit in one k8s cluster across two datacenters (Hetzner, with whom we couldn't be happier).

[GordonS]

Really interested in hosting at Hetzner, as their prices are fantastic by comparison to AWS, Azure & GCP.

I'm particularly interested in what an HA Postgres setup might look like. Assuming you are running some kind of database (whether Postgres or otherwise), what are you doing for persistent storage? Are you using Hetzner's cloud block storage volumes? What is performance like?

[jb_gericke]

Interesting! Is that a single K8 control plane across one cluster? We've gone with fully isolated clusters across 2 data centers to protect against a network isolation incident between them causing a split brain/borking etcd.

[zumachase]

Yes the control plane is only in one of the data centers. The other only runs admin services like offsite backups, our development infra (gitlab, etc) and CI/CD.

We could definitely do two clusters and probably should, but the secondary data center has few services that it wasn’t really worth the extra work.

[jb_gericke]

Oh cool, interesting. Thanks for the overview