[kamfc]

It made billions for companies. The rule of thumb is to start off "legal", build a big audience with huge benefits to the companies you're burrowing the data from, then quickly shift to legal. If you're not fast enough, you're dead. Web scraping and crawling has become less acceptable because the market leaders carved out the niches before you in a very "legal" way, and DO NOT want you as a competitor.