Hacker News new | ask | show | jobs
by ideonode 1540 days ago
Wasn't the inspiration for Hadoop not just a web crawler use-case, but also Google's famous MapReduce paper?
1 comments

Yes and no. The goal of the Nutch project was simply to create a web crawler, but it hit some scalability limits. Since Google had recently published two papers (MapReduce and Google Filesystem) that were quite relevant to scaling data processing and storage for a web crawler, Doug and Mike created an open source implementation of those ideas and redesigned the web crawler to use it.

The technology had many applications beyond a web crawler, of course, but that was the original use case.