Hacker News new | ask | show | jobs
by avip 2943 days ago
Google indexers respect robots.txt, so there goes the irony.
2 comments
p49k 2942 days ago
They also provide something of value to the operators of the websites they scrape, namely search traffic.
link
pbhjpbhj 2943 days ago
I've heard that they [sometimes] visit but don't index, is that true?
link