[winterbourne]

> If Google cuts that out completely, what incentive do websites have to not block the Google crawlers?

Completely, yes, that destroys the incentive. But they can reduce it 80% or 90% or so, to the point that it's just barely worthwhile to allow their crawlers.

[AnthonyMouse]

Suppose right now there are people making e.g. $60,000/year from their small site, or the same amount as a contributor to a medium-to-large site. If you take 90% of that, now they make $6000/year, which isn't enough to make a living, so instead they go take a job as a construction worker or a nurse or something, and then you're getting 90% of $0.

[winterbourne]

That's true, those numbers don't work out for Google. But they have essentially unlimited resources to discover the exact threshold at which that person is just barely incentivized to keep their site active. $100K/year, reduced 80% to $20K/year? Still enough for them to keep their site up part-time? Etc.

The bulk of the traffic they're referring is essentially residual profitless goodwill left over from their "don't be evil" days.