|
|
|
|
|
|
by codemusings
517 days ago
|
|
|
For what it's worth: they do honor the robots.txt file. I had the same problem with a client's CMS and denying all AI crawler user agents did the trick. It's clear they've all gone mad. The traffic spiked 400% overnight and made the CMS unresponsive a few times a day. |
|
|