[billfor]

I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.

[Bender]

load balancer / proxy, since you need a way to get the real ip

If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.

[inigyou]

And it's almost certainly exploitable by a client sending that same header.

[Bender]

Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.

[inigyou]

Not reliably enough. Look for parser differentials.

[63stack]

Please enlighten us

[inigyou]

Differences in parsing between two systems can allow data to sneak through. For example you might put a space before the header name. The reverse proxy might treat it as a header name starting with a space, so not matching the one it wants to remove, and not removed, while the origin server trims the space.

This is a ubiquitous security concern, like IDOR and memory safety. Every time you have a separate validation step for anything, you have the possibility of a parser differential vulnerability.

If you don't know it yet, look up "HTTP request smuggling" - pretty interesting.