[ToucanLoucan]

This rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.

[p3rls]

I think my team has spent about a quarter of this year's dev time on defending against scrapers. Their arrogance is the stuff of ancient greek dramas.