Hacker News new | ask | show | jobs
by wesley 4971 days ago
Wouldn't a proper robots.txt rule fix this?
1 comments

A robots.txt file disallowing crawling on the sites that display the contents of user email would help fix this.

However, as some of the discussion below points out, I don't believe that disallowing crawling of these URLs in our robots.txt would keep them from the index if a search engine finds reference to them elsewhere; I think it simply keeps them from being crawled.