Hacker News new | ask | show | jobs
A web crawler that can handle any number of websites (mixnode.com)
17 points by mixnode 3350 days ago
1 comments
spoondocz 3349 days ago
What's the largest crawl you can handle? meaning your upper limit...
link
mixnode 3349 days ago
We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.
link