|
|
|
|
|
|
by tomwheeler
1540 days ago
|
|
|
Yes and no. The goal of the Nutch project was simply to create a web crawler, but it hit some scalability limits. Since Google had recently published two papers (MapReduce and Google Filesystem) that were quite relevant to scaling data processing and storage for a web crawler, Doug and Mike created an open source implementation of those ideas and redesigned the web crawler to use it. The technology had many applications beyond a web crawler, of course, but that was the original use case. |
|
|