|
|
|
|
|
by dorgo
2498 days ago
|
|
>"more what you call guidelines, than actual rules" they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google. |
|
Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.