Y
Hacker News
new
|
ask
|
show
|
jobs
by
ikeboy
3721 days ago
Doesn't excluding them in robots.txt solve the search engine problem?
1 comments
_Understated_
3721 days ago
Following the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
link
ikeboy
3721 days ago
The dodgy ones don't have access to the URLs. The example was Skype links: if Microsoft scrapes those, they'll follow the rules and not make them available in searches.
If you give your link to a dodgy search engine, you've lost.
link