Why are you assuming that a determined attacker doesn't control your L4 stack? MITMs are a threat, your network could be compromised, routers (especially consumer routers) are rife with vulnerabilities. This is the entire reason "zero trust" is pushed.
In any serious security design, "the attacker probably won't do that" would and should be shot down immediately. If your security strategy is hoping that an attacker will be kind enough to not exploit your open vulnerability, you've already failed at threat modeling and at security.
If an attacker can do it, you must assume they will do it. Because they will. That should be the starting point for any threat model.
that's cool man, i'm still going to block the 99.9999% of attackers that don't own my isp. you are conflating "bad idea in extremely exotic scenario" with "counterproductive"; ever heard of defense in depth?