[stanferder]

I don't think the direction would ever be clear, even if the legality were clearly established. The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field.

Companies want to provide some information to some people; but providing all information to all people is analogous to allowing customers to make a meal of free food samples, on a recurring basis.

[mindcrime]

The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field.

It already is. There are entire companies, like Distil Networks, who exist solely to protect companies from bots/scrapers/etc. Actually, looks like Distil got acquired and are now part of Imperva, but anyway, the idea is the same. This is definitely an existing field.

https://www.imperva.com/products/bot-management/

Disclosure: former Distil employee, but I have no financial stake in this discussion, and have mixed feelings about scraping. Clearly it can be beneficial in some situations, but when I think about having to pay exorbitant prices to scalpers for tickets to an event, because they used a bot to buy up all the tickets, that is less appealing.