|
|
|
|
|
|
by epc
446 days ago
|
|
|
I’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it. I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network. |
|
|