|
|
|
|
|
|
by hacsky
1401 days ago
|
|
|
I think robots.txt needs to work the other way. You put in the robots.txt ONLY if you want the search engines to crawl your page. The current way is akin to saying ... "You do not have a lock on your door. So we are welcome to come in ..." |
|
|
Robots.txt is a way of saying "these are not the pages you are looking for".