|
|
|
|
|
|
by blauwbilgorgel
4841 days ago
|
|
|
Incorrect. To configure your robots.txt to not be crawled at all use: User-agent: *
Disallow: /
Allow indexing of everything with: User-agent: *
Disallow:
It seems they are at this very moment struggling to change things. |
|
|