|
|
|
|
|
|
by chasd00
349 days ago
|
|
|
a crawler doesn't have to change anything, they can just ignore the robots.txt file. It's up to the client to read robots.txt and follow its directives but there's no technical reason why the client cannot just ignore everything in the file period. |
|
|