Could you indulge me with any details? Not AWS or someone who works on k8s, I'm just always interested in war stories, and cloud providers "not working" is an interest too.
We got an on-prem cluster working with Rancher and MetalLB with L2 configuration (MetalLB spoke ARP). It was great.
We tried to do the same on EKS. we got it iniitally working, but then we needed to get the nodes on certain subnets so that HQ would be happy or something, that was hard. It felt like any time we tried to make any changes, such as adding worker nodes or deleting old ones, the ALB configuration got all messed up and we couldn't get it fixed. Super brittle. We tried with NLB too, but eventually just got tired of fighting with a tool that was supposed to make this stuff simple. I'll tell you what was simple: MetalLB on our own on-prem network.