Hacker News new | ask | show | jobs
by vinceguidry 2945 days ago
Google's web scrapers obey robots.txt, you can stop Google from crawling your website if you want. Google doesn't want you crawling their website.

That word, I don't think it means what you think it means.