|
|
|
|
|
|
by swatcoder
804 days ago
|
|
|
The convention is that crawlers first read /robots.txt to see what they're encouraged to scrape and what they're not meant to, and then hopefully honor those directions. In this case, as in many, the disallow rules are intentionally meant to protect the signal quality and efficiency of the crawler. |
|
|