| Edit: on reading a few more comments, I think this is probably all wrong... The TLS Client hello is not that big (the client sent FIN is seq=518), and the server is only sending packets with SEQ=0. As others pointed out this likely means that the server that received the SYNs is not receiving the final ACK and data packets. From what I can tell, the example IP is not broadly anycast. From my test hosts in Seattle, traceroute takes me trhough transit to San Jose, and then either vl201.sjc-eq10-dist-1.cdn77.com or vl202.sjc-eq10-dist-1.cdn77.com and finally 169-150-221-147.bunnyinfra.net I'm not sure how easy it is to run a traceroute with tcp with different flags. But if the OP can run a traceroute with only the SYN flag, and again with only the ACK flag, that might be pretty interesting. I suspect this is an issue inside BunnyCDN's network where packets from this user/network with SYN go to one server host, and with ACK go to another. Maybe there's an odd router somewhere that's routing these differently, but if they both make it to Bunny, they should both work. With $ traceroute --version
Modern traceroute for Linux, version 2.1.2
Copyright (c) 2016 Dmitry Butskoy, License: GPL v2 or any later
I can specify to do a traceroute with syn or ack with traceroute 169.150.221.147 -p 443 -q 1 -T -O ack
or traceroute 169.150.221.147 -p 443 -q 1 -T -O syn
Wrong answer about MTU below for posterity:Yeah, that would be my bet too. Especially with a after 60 seconds things start to work, I think that's the timeout for windows to do PMTU Blackhole probing (which is painfully slow; iOS and I think MacOS do it much sooner; I think even Android has gotten around to doing it in a reasonable amount of time) I've got a test site up that might work for the OP http://pmtud.enslaves.us/ But, if it's really only happening with BunnyCDN, it's possible that most of their routes are 1500 MTU clean (or have working path MTU) and only the routes to get to BunnyCDN aren't. Of course, a lot of popular services intentionally drop their advertised MTU and allowed outbound MTU to work around the many broken networks out there, so service X and Y works doesn't really mean the path is clean. |
I had seen this exact issue with Fastly a few years ago.