|
|
|
|
|
|
by otherme123
428 days ago
|
|
|
This is my experience with AI bots. This is my robots.txt: User-agent: *
Crawl-Delay: 20 Clear enough. Google, Bing and others respect the limits, and while about half my traffic are bots, they never DoS the site. When a very well known AI bot crawled my site in august, they fired up everything: fail2ban put them temporarily in jail multiple times, the nginx request limit per ip was serving 426 and 444 to more than half their requests (but they kept hammering the same Urls), and some human users contacted me complaining about the site going 503. I had to block the bot IPs at the firewall. They ignore (if they even read) the robots.txt. |
|
|