[vbezhenar]

The question is: who has the leverage?

If some small news website denies Google Bot crawling, it'll disappear from Google and essentially it'll disappear from the Internet. People do a great lengths to appease the Google Crawler.

If some huge news website demands fees from Google, it might work, I guess. But I'm not sure that it would work even for BBC or CNN.

[jimbohn]

I agree about the leverage and small website reasoning, definitely some game-theory related thinking is needed to get something like this right. But it does feel like this enables the "unionization" of websites against scraping giants, google is in an especially interesting position because, as you mentioned, could blackmail you into scraping in exchange for indexing.

[ipaddr]

If its a smaller news site they have already de-ranked them, and used their content for AI answers