Hacker News new | ask | show | jobs
by bhartzer 2309 days ago
Google will index pages disallowed in robots.txt.

If you want a page out of the index, you must allow crawling in robots.txt and use noindex on the page.

It used to be that you could use robots.txt to stop indexing but google changed the rules a whole back.