[throwawaysoxjje]

Scraping for search engines is different. They need to scrape the data to build the index, otherwise the search wouldn’t work.

[crote]

Being indexed in a search engine benefits the website owner, as it drives traffic towards the website.

Being used as AI training data provides negative value for a website owner, as it takes traffic away.

It's the difference between a movie review, and a ripped torrent.