Hacker News new | ask | show | jobs
by toast0 813 days ago
Edit: on reading a few more comments, I think this is probably all wrong...

The TLS Client hello is not that big (the client sent FIN is seq=518), and the server is only sending packets with SEQ=0. As others pointed out this likely means that the server that received the SYNs is not receiving the final ACK and data packets.

From what I can tell, the example IP is not broadly anycast. From my test hosts in Seattle, traceroute takes me trhough transit to San Jose, and then either

vl201.sjc-eq10-dist-1.cdn77.com or vl202.sjc-eq10-dist-1.cdn77.com and finally

169-150-221-147.bunnyinfra.net

I'm not sure how easy it is to run a traceroute with tcp with different flags. But if the OP can run a traceroute with only the SYN flag, and again with only the ACK flag, that might be pretty interesting. I suspect this is an issue inside BunnyCDN's network where packets from this user/network with SYN go to one server host, and with ACK go to another. Maybe there's an odd router somewhere that's routing these differently, but if they both make it to Bunny, they should both work.

With

    $ traceroute --version
    Modern traceroute for Linux, version 2.1.2
    Copyright (c) 2016  Dmitry Butskoy,   License: GPL v2 or any later
I can specify to do a traceroute with syn or ack with

     traceroute 169.150.221.147 -p 443 -q 1 -T -O ack
or

     traceroute 169.150.221.147 -p 443 -q 1 -T -O syn
Wrong answer about MTU below for posterity:

Yeah, that would be my bet too. Especially with a after 60 seconds things start to work, I think that's the timeout for windows to do PMTU Blackhole probing (which is painfully slow; iOS and I think MacOS do it much sooner; I think even Android has gotten around to doing it in a reasonable amount of time)

I've got a test site up that might work for the OP http://pmtud.enslaves.us/

But, if it's really only happening with BunnyCDN, it's possible that most of their routes are 1500 MTU clean (or have working path MTU) and only the routes to get to BunnyCDN aren't. Of course, a lot of popular services intentionally drop their advertised MTU and allowed outbound MTU to work around the many broken networks out there, so service X and Y works doesn't really mean the path is clean.

1 comments

ClientHello isn't that big but ServerHello that's in the reply can be quite large and since TCP packets have DF flag set, some middleware box may toss it if PMTUD didn't work correctly.

I had seen this exact issue with Fastly a few years ago.

Yeah, I expected a large ServerHello, but then I would expect the server to send Seq=[LargeNumber] packets. Often you'd get an ACK for the ClientHello, then a missed packet or several, then the final packet of the ServerHello which is often small. Or at least an ack from the resend of ClientHello with a large sequence number.

I guess I've seen pmtud issues way too often in my life, and I just jumped ahead. :D