Hacker News new | ask | show | jobs
by polaco 4045 days ago
Yes. This need to be emphasized: Disallowing URLs in robots.txt will not necessarily exclude them from search results. Search engines will still find those pages if they are linked to or mentioned somewhere else.

The search result will consist only of the URL, and the snippet will say "A description for this result is not available because of this site's robots.txt"

Use the noindex tag, folks. Also, Google Webmaster Tools allows you to remove URLs from Google's index.

Ref.: https://yoast.com/prevent-site-being-indexed/