Hacker News new | ask | show | jobs
by mdaniel 4149 days ago
As I mentioned in a sibling reply, legality depends on your jurisdiction but in the US AFAIK scraping is no different from connecting to LinkedIn 15,000,000 times with a copy of Chrome.

To further illustrate this point, LinkedIn provides a helpful directory at the bottom of their homepage, allowing one to browse every user[0] and every company[1] in their graph. Hard pressed to argue that it's illegal to navigate to every single profile if you serve a directory of them.

Also, I believe there are companies[2] and shops[3] that specialize in doing that kind of thing. I would say it would be worth reaching out to them, if nothing else just to get a sense for what something like that would cost.

0 = https://www.linkedin.com/directory/people-1-5-1/ 1 = https://www.linkedin.com/directory/companies 2 = http://scrapinghub.com/ (makers of the outstanding Scrapy framework!) 3 = http://webscrapinghelp.com