|
|
|
|
|
|
by Kadrith
5071 days ago
|
|
|
I know, but I thought the existence of robots.txt was why Google is allowed to crawl sites. If a site disagrees with the crawling they can add a robots.txt entry and Google will honor it. It at least shows that you are giving the publisher an option. |
|
|