[throwawayffffas]

While I agree with you on sentiment, bandwidth and the compute that power the websites these scrapers cost money, most people that have websites cannot afford to provision the capacity required to serve these requests.

Additionally the objective of these pages is to serve ads to real people or funnel real people to paid products, AI traffic for them is just a cost at best and a denial of service attack at worst.