| HN Mirror

Y	Hacker News new \| ask \| show \| jobs

by theomega 1367 days ago

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work:

  $ curl -v https://fly.io/blog/introducing-litefs/
  *   Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer

Requesting via ipv4 works

  $ curl -4v https://fly.io/blog/introducing-litefs/
  *   Trying 37.16.18.81:443...
  * Connected to fly.io (37.16.18.81) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  * (304) (IN), TLS handshake, Server hello (2):
  * (304) (IN), TLS handshake, Unknown (8):
  * (304) (IN), TLS handshake, Certificate (11):
  * (304) (IN), TLS handshake, CERT verify (15):
  * (304) (IN), TLS handshake, Finished (20):
  * (304) (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=fly.io
  *  start date: Jul 25 11:20:01 2022 GMT
  *  expire date: Oct 23 11:20:00 2022 GMT
  *  subjectAltName: host "fly.io" matched cert's "fly.io"
  *  issuer: C=US; O=Let's Encrypt; CN=R3
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multiplexing
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x135011c00)
  > GET /blog/introducing-litefs/ HTTP/2
  > Host: fly.io
  > user-agent: curl/7.79.1
  > accept: */*
  >
  * Connection state changed (MAX_CONCURRENT_STREAMS == 32)!
  < HTTP/2 200
  < accept-ranges: bytes
  < cache-control: max-age=0, private, must-revalidate
  < content-type: text/html
  < date: Wed, 21 Sep 2022 16:50:16 GMT
  < etag: "632b20f0-1bdc1"
  < fly-request-id: 01GDGFA3RPZPRDV9M3AQ3159ZK-fra
  < last-modified: Wed, 21 Sep 2022 14:34:24 GMT
  < server: Fly/51ee4ef9 (2022-09-20)
  < via: 1.1 fly.io, 2 fly.io
  <
  <!doctype html> ...

3 comments

toast0 1367 days ago

Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

link

ignoramous 1367 days ago

> Could easily be a path MTU issue?

Perhaps: https://community.fly.io/t/ipv6-pmtud-issue/5081

link

theomega 1367 days ago

How could I find out if this is the case? And, what can I do about it? Disabling IPv6 obviously fixes it, but that it not a solution...

link

toast0 1367 days ago

I'm not aware of a simple to use test site for this on IPv6 (I've got one for IPv4, but my server host doesn't do IPv6 either, and using a tunnel will limit the ranges I can test, etc)... so you'll need to test ye olde fashioned way.

I don't have a mac, but if the man page[1] is right, something like this should work to see how big of a packet you can successfully send and receive:

      ping -6 -D -G 1500 -g 1400 fly.io

(you may need to run as root to set packet sizes). You should get back a list of replies with say 1408 bytes, then 1409, etc. The last number you get back is effectively the largest IPv6 payload you can receive (on this path, it could be different for other paths), and if you add the IPv6 header length of 40, that's your effective path MTU.

Use tcpdump to see what TCP MSS is being sent on your outgoing SYN packets, for IPv4, the MSS is MTU - 40 (20 for IPV4 header, 20 for TCP header), for IPv6, the MSS should be MTU - 60 (40 for IPv6 header, 20 for TCP header). If your TCP MSS is higher than the observed path MTU to fly.io, that's likely the immediate cause of your problem.

If you're using a router, make sure it knows the proper MTU for the IPv6 connection, and enable MSS clamping on IPv6, if possible --- or make sure the router advertisement daemon shares the correct MTU.

Hope this gets you started.

[1] https://ss64.com/osx/ping.html

link

theomega 1367 days ago

Thanks, that was of great help!

Just for reference, the ping command is a little different

  sudo ping6 -D -G 1500,1400 fly.io

I set an MSS to 1492 which pfsense (my router) translates to an MSS clamp of 1492-60 for IPv6 and 1492-40 for IPv4. This is a German Deutsche Telekom Fiber connection. Now everything works fine, I can request fly.io (and also discovered that https://ipv6-test.com was not working before and now does with the MSS clamping)

Does MSS clamping have any disadvantages? Are there any alternatives in my case?

link

toast0 1367 days ago

Excellent, happy to help, glad you figured out the right command!. 1492 MTU is consistent with PPPoE, or a network where they didn't want to run a separate MTU for PPPoE and straight ethernet.

The only downside to MSS clamping is the computational expense of inspecting and modifying the packets. On a residential connection, where you're running pfsense already, it's probably not even noticeable; but your ISP wouldn't be able to do clamping for you, because large scale routers don't have the processing budget to inspect packets at that level. I've seen some MSS clamping implementations that only clamp packets going out to the internet, and not the return packets... that can lead to problems sending large packets (which isn't always very noticeable, actually; a lot of basic browsing doesn't send packets large enough to hit this, unless you go to a site that sets huge cookies or do some real uploading)

The alternative would be to run a 1492 MTU on your LAN, but that has the marginal negative of reducing your maximum packet size for LAN to LAN transfers.

link

wglb 1367 days ago

Just tried it on my linux box:

  * Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) ALPN,
  * offering h2 ALPN, offering http/1.1 CAfile:
  * /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs TLSv1.0
  * (OUT), TLS header, Certificate Status (22): TLSv1.3 (OUT), TLS
  * handshake, Client hello (1): TLSv1.2 (IN), TLS header, Certificate
  * Status (22): TLSv1.3 (IN), TLS handshake, Server hello (2): TLSv1.2
  * (IN), TLS header, Finished (20): TLSv1.2 (IN), TLS header,
  * Supplemental data (23): TLSv1.3 (IN), TLS handshake, Encrypted
  * Extensions (8): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Certificate (11): TLSv1.2 (IN), TLS a
  * header, Supplemental data (23): TLSv1.3 (IN), TLS handshake, CERT
  * verify (15): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Finished (20): TLSv1.2 (OUT), TLS
  * header, Finished (20): TLSv1.3 (OUT), TLS change cipher, Change
  * cipher spec (1): TLSv1.2 (OUT), TLS header, Supplemental data (23):
  * TLSv1.3 (OUT), TLS handshake, Finished (20): SSL connection using
  * TLSv1.3 / TLS_AES_256_GCM_SHA384 ALPN, server accepted to use h2
  * Server certificate:

...

Same on my macbook

link

tptacek 1367 days ago

Works fine over IPv6 from this Mac.

link