Hacker News new | ask | show | jobs
by spimmy 4695 days ago
What do you mean by "open"? Can the data be used for startups and other commercial purposes?
3 comments

Yes! Startups/commercial companies/etc can all use the data for free. The terms of use basically say, don't do anything illegal with it and a few other things, but it shouldn't affect the vast majority of uses.

Actually, tomorrow a video on a startup that uses Common Crawl data is getting posted.

From the FAQ: "Please refer to the Common Crawl Terms of Use document for a detailed, authoritative description of our Terms of Use guidelines, but, in general, you cannot republish the data retrieved from the crawl (unless allowed by fair use), you cannot resell access to the service, you cannot use the crawl data for any illegal purposes, and you must respect the Terms of Use of the sites we crawl."

http://commoncrawl.org/about/terms-of-use/

The data is freely available: http://aws.amazon.com/datasets/41740

and you just need to comply with the Common Crawl TOU: http://commoncrawl.org/about/terms-of-use/