Hacker News new | ask | show | jobs
by MCorbani 2468 days ago
Agreed, some websites are really reticent about scraping. But let's think about Google, they are scraping the whole web regardless of the ToS of the websites, so it all boils down to one question : do you create value for the website owner ? That's why we want to focus on use-cases where we create value for both, our users and the website owner. If you think about Yodlee/Plaid in the banking sector, they built partnerships with the Banks but continued scraping them because most of them didn't provide an API.
1 comments

Google respects robots.txt.