[wslh]

> People who ignore polite requests are assholes, and we are well within our rights to complain about them.

If you are building a new search engine and the robots.txt only include Google, are you an asshole indexing the information?

[kijin]

Yes, because the site owner has clearly and explicitly requested that you don't scrape their site, fully accepting the consequence that their site will not appear in any search engine other than Google.

Whatever impact your new search engine or LLM might have in the world is irrelevant to their wishes.