[dilyevsky]

They also charge you like 1c/GB for traffic egress between the zones. To top it off there are issues with AWS loadbalancers in multi-zone setups. Ultimately i've come to the conclusion that large multi-zonal clusters is a mistake. Do several single-zone disposable clusters if you want zone redundancy.

[frenchtoast8]

At $WORK traffic between zones ($REGION-DataTransfer-Regional-Bytes) is our second largest cost on our AWS bill, more than our EC2/EKS cost. It adds up to mid six figures each year. We try to minimize this where it is easy to do so. For example, our EKS pods perform reads against RDS read replicas in the same AZ only, but you're out of luck for writes to the primary instance. To reduce this in any significant way can eat up a lot of time, and for us, the cost is enough to be painful but not enough to dedicate an engineer to fixing.

This is precisely how Amazon's bread is buttered. An outage affecting an entire AZ is rare enough that I would feel pretty happy making all our clusters single-AZ, but it would be a fool's errand for me to convince management to go against Amazon's official recommendations.

[toredash]

I would LOVE to pitch something else I'm working on that is solving this problem in EKS, cross zone data transfer.

It's a plugin that enables traffic re-direction for any service that is using an IP in any given VPC. If you have say multiple RDS Reader instances, it will first attempt to use local AZ instances first, but the other instances are available if local instances are non-functional. So you do not loose HA or failover features.

The plugin does not require any reconfiguration on your apps. It works similar to Topology Aware Routing (https://kubernetes.io/docs/concepts/services-networking/topo...) in Kubernetes, but it works for services outside of Kubernetes. The plugin even works for non-Kubernetes setup as well.

This AZP solution is fine for services that is have one IP or primary instance, like RDS Writer instance. It does not work for anything that is "stateless" and multi-AZ, like RDS Read-only instances or ALBs.

[dilyevsky]

I assume with this much traffic you’re running multiple clusters? In that case what is there to gain by running each cluster as multi-zone?