[OutOfHere]

It is harmful because innocent users routinely get caught in your dragnet. And why even have a public website if the goal is not to serve it?

What is the actual problem with serving users? You mentioned incredible load. I would stop using inefficient PHP or JavaScript or Ruby for web servers. I would use Go or Rust or a comparable efficient server with native concurrency. Survival always requires adaptation.

How do you know that the alleged proxies belong to the same scrapers? I would look carefully at the values contained in the IP chain as determined by XFF to know which subnets to rate-limit as per their membership in the XFF.

Another way is to require authentication for expensive endpoints.