|
|
|
|
|
|
by cmatthias
1256 days ago
|
|
|
After reading this and your response to a sibling comment I wholeheartedly disagree with you on both the specific definition of the word crawler and what the "main purpose" of robots.txt is, but glad we can agree that Google should be doing more to respect rate limits :) |
|
|