[css]

how do you avoid getting banned by the companies you scrape? Most ToSs have a clause like:

> We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means... [etc]

[reascenda]

This may now be moot after the LinkedIn Vs hiq labs case a couple of days ago which appears to have blanket legalised we scraping.

[kube-system]

hiQ v. LinkedIn means you probably aren't going to jail for scraping LinkedIn's website. It doesn't mean LinkedIn can't IP ban you.

[MCorbani]

Agreed, some websites are really reticent about scraping. But let's think about Google, they are scraping the whole web regardless of the ToS of the websites, so it all boils down to one question : do you create value for the website owner ? That's why we want to focus on use-cases where we create value for both, our users and the website owner. If you think about Yodlee/Plaid in the banking sector, they built partnerships with the Banks but continued scraping them because most of them didn't provide an API.

[kube-system]

Google respects robots.txt.

[bytematic]

https://www.theverge.com/2017/8/15/16148250/microsoft-linked...

[AznHisoka]

BYOP (bring your own proxies)