[buzer]

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google.

They cannot. Googlebot & some other search engine bots (like Bing's & Yandex's) get special treatment in various websites. This includes things like ban on non-whitelisted scrapers & bypassing paywalls. If you are not already established player in the field, you would not get able to scrape the same websites as the established players can.

[bmpafa]

As I understand it, this was the rationale behind the courts' decision to prohibit LinkedIn from banning people from scraping public profiles.

Basically it was anti competitive to grant certain privileges to major players around 'public data,' but to block smaller players.

No telling if/when ramifications from that decision (last year) hit existing anti scraping measures, though.