Hacker News new | ask | show | jobs
by ikeboy 3721 days ago
Doesn't excluding them in robots.txt solve the search engine problem?
1 comments

Following the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
The dodgy ones don't have access to the URLs. The example was Skype links: if Microsoft scrapes those, they'll follow the rules and not make them available in searches.

If you give your link to a dodgy search engine, you've lost.