Hacker News new | ask | show | jobs
by patio11 6346 days ago
spelled out like this so Google won't index it

Obscurity will NOT HELP YOU avoiding a Google index. They have many sources of data, including receiving toolbar data from users, and they very pointedly do not mention all sources they use to generate the crawl lists.

If you want to not get indexed, use the nonindex meta tag, or sign up for their webmaster console and remove that particular URL from the index. (Somewhat counterintuitively, robots.txt-ing out a site doesn't prevent it from being indexed, only from being crawled. They will still include it in their search results if it has external indicia of trust.)