|
|
|
|
|
|
by blyry
2167 days ago
|
|
|
I asked internally and figured out the gotcha that bit us: default dns payload size is 512b, which is enough for a few backend hosts but for sure not 12 or 30. Limit is 8kb, which probably wouldn't work for whatever slack is doing. https://cbonte.github.io/haproxy-dconv/2.1/configuration.htm.... Because DNS records come back in random order for each response, those truncated dns responses caused the backend slots to constantly rotate between different pod instances. Haproxy was graceful about the rotations, but it showed up as suddenly very strange latency / perf numbers when a backend was scaled up to say 10 instances from the normal 3 |
|
|