[righthand]

It’s not dumb because Googlebot follows the robots.txt rules. That is the sincere crux of it all. No one is going to casually open their site up to Llms that are blatantly scraping their site to then use their information to displace them.

Not blocking violent, bad-actor scrapers is dumb. Letting through bad-actor scrapers because a bunch of rich people want to make it the norm is dumb.

Llms are not directing traffic to the sites and that is the tradeoff that site owners allow with Googlebot. Even if Perplexity or Claude will provide a source, the Llm user is most likely not asking/clicking for it 99% of the time.