[LorenPechtel]

When most of your server capacity is going to answering the scrapers it matters. It's not that the stuff is hidden, it's that storefront being flooded with 10x as many customers as the fire code allows. And some of them go around asking your employees mindless questions. (Small forum I help moderate: we were getting hammered with what was probably some sort of AI that was taking search queries and feeding them into the forum search. Search is now registered users only.)

[reconnecting]

> When most of your server capacity is going to answering the scrapers it matters

I've been dealing with the web since the previous century and still haven't managed to build a website that could be hurt by scrapers visiting it.

If you went through the logs, you'd probably see that these bots are on a single IP or subnet, which can be easily detected and blocked instead of closing off search to non-registered users.

[trumpdong]

That's incorrect, they use residential proxy networks.

[LorenPechtel]

Botnet.

Our offending searches were coming from many addresses.