Hacker News new | ask | show | jobs
Lessons Learned While Crawling the Web (moz.com)
33 points by jennita 4648 days ago
1 comments
alecsmart1 4648 days ago
The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"
link
jgmmo 4648 days ago
I think it should be called 'web scraping 101'

Very thin article.

link
billrobertson42 4647 days ago
I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.
link