Ask HN: Crawl the entire web?

Y	Hacker News new \| ask \| show \| jobs

	Ask HN: Crawl the entire web?
	2 points by nullundefined 3159 days ago
	If you were tasked with crawling the entire web, how would you go about doing it?

1 comments

I'd grab the CommonCrawl data and keep moving on what my actual problem is.