Hacker News new | ask | show | jobs
by mdaniel 4149 days ago
For free or for money?

I believe there are web scraping companies that sell monthly dumps of LinkedIn data, but I don't know off-hand how much something like that would cost.

It would also likely depend on whether you wanted US-only data or worldwide.
2 comments
ninjaranter 4149 days ago
US-only is fine. For money is potentially fine (I've no idea how much something like that would cost), if I knew it was legal - though if there was a legal way to get that data through scraping I'd probably just do that myself :)
link
mdaniel 4149 days ago
As I mentioned in a sibling reply, legality depends on your jurisdiction but in the US AFAIK scraping is no different from connecting to LinkedIn 15,000,000 times with a copy of Chrome.

To further illustrate this point, LinkedIn provides a helpful directory at the bottom of their homepage, allowing one to browse every user[0] and every company[1] in their graph. Hard pressed to argue that it's illegal to navigate to every single profile if you serve a directory of them.

Also, I believe there are companies[2] and shops[3] that specialize in doing that kind of thing. I would say it would be worth reaching out to them, if nothing else just to get a sense for what something like that would cost.

0 = https://www.linkedin.com/directory/people-1-5-1/ 1 = https://www.linkedin.com/directory/companies 2 = http://scrapinghub.com/ (makers of the outstanding Scrapy framework!) 3 = http://webscrapinghelp.com

link
anabranch 4149 days ago
Is this legal to scrape linkedin? What's the legality of obtaining + selling this information?
link
mdaniel 4149 days ago
AFAIK, scraping is legal under US law because one does not have to circumvent any encryption to access the webpage. If LinkedIn will serve it to Google, Bing and others, then they'd be hard pressed to say "our HTML is secret."

Selling it is a different matter, and one that I don't have enough law background to answer. It may fall under the provision in US copyright law that says that facts are not copyrightable. So long as you are just selling the facts of the graph that they gladly serve to the Internet, ... again, I am not experienced enough to comment.

link