Y
Hacker News
new
|
ask
|
show
|
jobs
by
jumperjake
3893 days ago
Not correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.
1 comments
chinathrow
3893 days ago
They do not index the content but might add the URL, correct. You can have a meta noindex present and they won't index even the URL.
link