Hacker News new | ask | show | jobs
by xagarth 3400 days ago
Crawling described here is very inefficient. For efficient and high performance crawling I recommend libcurl and curlmulti.