Hacker News new | ask | show | jobs
by dickeytk 2308 days ago
nofollow is for external links, robots.txt is for pages on your own domain. They don’t reference the same thing.
1 comments

Google will index pages disallowed in robots.txt.

If you want a page out of the index, you must allow crawling in robots.txt and use noindex on the page.

It used to be that you could use robots.txt to stop indexing but google changed the rules a whole back.