Hacker News new | ask | show | jobs
by SoftwareMaven 5405 days ago
That's what robots.txt is for, not silly captchas. You put captchas for the people who ignore that file.
2 comments

I think the idea is that given a large corpus of filtered and unfiltered reviews, you might be able to reverse engineer signals in the algorithm and game the system. If that's your end goal, you and the software you write is likely to ignore robots.txt directives.
Not all spiders honor robots.txt