[Existing4190]

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall

I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

[Thev00d00]

Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.

[axus]

They should go to http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.

[WirelessGigabit]

That's weird. I get nothing there.

[js2]

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

[xnyan]

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

[js2]

> BTW your assumption

I did not assume. The comment to which I was responding suggested it was the destination IP that was the problem. Generally (but not always) an IP filter would be applied irrespective of protocol. I also pointed out that the initial SYN and reply SYN/ACK are getting through the hypothesized bogon filter and those are part of TCP. I don't think the bogon filter is a hypothesis that fits the evidence.

ETA: but adding connection state tracking + a filter does make sense https://news.ycombinator.com/item?id=39822214

[xnyan]

> Generally (but not always) an IP filter would be applied irrespective of protocol

I've never seen this (all protocols by default) in any environment I've personally worked with, but perhaps I've had a unique experience.

[whirlwin]

> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

Are you referring to local networks only?

It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

[pajko]

Fragmented packets won't work without ICMP.

Edit: here's a good page about the effects of disabling ICMP: https://www.rimscout.com/why-you-should-not-block-icmp/

Also there's some blackhole detection or how is it called.

However it's OK to block _parts_ of the ICMP protocol for security reasons, like echo and reply.

[fragmede]

That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.

[Hikikomori]

AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).

[immibis]

This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.

[xyst]

Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.

[js2]

That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.

[dunham]

It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped.

EDIT - I've now scrolled down in HN and saw that this was ruled out.

[Existing4190]

You are right. My comment can't solve the whole story.

Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

[xnyan]

Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.

[js2]

Filtering + established connection tracking:

https://news.ycombinator.com/item?id=39822214

[dunham]

My employer did something like that once, and it took out access to github.