[pona-a]

Volume and incentives.

Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who needs an index full of dead links. With LLM crawlers, all you need is text, and if the site is forced to shut down because of you, that's just less data for your competitors.

[throw10920]

Also, nobody else steals your stuff like AI does. Doesn't take much thought to figure out the difference.