[zarzavat]

Because often that data is only available through scraping.

Nobody wants to scrape, it's messy and fickle and a general pain in the backside. But sometimes the data you need exists only in that form.

If you run a website and you have a problem with scrapers, then make all that data available through an API and say what acceptable rate limits are. If cost is an issue, then charge a proportionate fee, my time writing a scraper is worth much more than paying a few dollars for an API.

If you just say "No" to everything then you lose all control over the process and the only outcome will be such an arm race.

[kingcharles]

God. This. The number of times I've spent 2 days of my very expensive time coding a scraper to get data I'll use once, when I would have paid a few dollars just to download it in a text file.