[toast0]

Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

[ignoramous]

> Could easily be a path MTU issue?

Perhaps: https://community.fly.io/t/ipv6-pmtud-issue/5081

[theomega]

How could I find out if this is the case? And, what can I do about it? Disabling IPv6 obviously fixes it, but that it not a solution...

[toast0]

I'm not aware of a simple to use test site for this on IPv6 (I've got one for IPv4, but my server host doesn't do IPv6 either, and using a tunnel will limit the ranges I can test, etc)... so you'll need to test ye olde fashioned way.

I don't have a mac, but if the man page[1] is right, something like this should work to see how big of a packet you can successfully send and receive:

      ping -6 -D -G 1500 -g 1400 fly.io

(you may need to run as root to set packet sizes). You should get back a list of replies with say 1408 bytes, then 1409, etc. The last number you get back is effectively the largest IPv6 payload you can receive (on this path, it could be different for other paths), and if you add the IPv6 header length of 40, that's your effective path MTU.

Use tcpdump to see what TCP MSS is being sent on your outgoing SYN packets, for IPv4, the MSS is MTU - 40 (20 for IPV4 header, 20 for TCP header), for IPv6, the MSS should be MTU - 60 (40 for IPv6 header, 20 for TCP header). If your TCP MSS is higher than the observed path MTU to fly.io, that's likely the immediate cause of your problem.

If you're using a router, make sure it knows the proper MTU for the IPv6 connection, and enable MSS clamping on IPv6, if possible --- or make sure the router advertisement daemon shares the correct MTU.

Hope this gets you started.

[1] https://ss64.com/osx/ping.html

[theomega]

Thanks, that was of great help!

Just for reference, the ping command is a little different

  sudo ping6 -D -G 1500,1400 fly.io

I set an MSS to 1492 which pfsense (my router) translates to an MSS clamp of 1492-60 for IPv6 and 1492-40 for IPv4. This is a German Deutsche Telekom Fiber connection. Now everything works fine, I can request fly.io (and also discovered that https://ipv6-test.com was not working before and now does with the MSS clamping)

Does MSS clamping have any disadvantages? Are there any alternatives in my case?

[toast0]

Excellent, happy to help, glad you figured out the right command!. 1492 MTU is consistent with PPPoE, or a network where they didn't want to run a separate MTU for PPPoE and straight ethernet.

The only downside to MSS clamping is the computational expense of inspecting and modifying the packets. On a residential connection, where you're running pfsense already, it's probably not even noticeable; but your ISP wouldn't be able to do clamping for you, because large scale routers don't have the processing budget to inspect packets at that level. I've seen some MSS clamping implementations that only clamp packets going out to the internet, and not the return packets... that can lead to problems sending large packets (which isn't always very noticeable, actually; a lot of basic browsing doesn't send packets large enough to hit this, unless you go to a site that sets huge cookies or do some real uploading)

The alternative would be to run a 1492 MTU on your LAN, but that has the marginal negative of reducing your maximum packet size for LAN to LAN transfers.