[brudgers]

My guess is they see correlation between patterns of automated access and problematic behaviors.

In other words, the current state is the result of hard won experience not syllogistic reasoning.

[ksajadi]

That is a good hypothesis but then you’d imagine they could combine it with registered clients to keep crawlers accountable.

[brudgers]

‘Keeping crawlers accountable’ sounds like whack-a-mole.

At least to me because benign crawling is likely to be a tiny subset of all crawling activity that is not done by google, etc.