|
|
|
|
|
|
by sparkling
2 hours ago
|
|
|
Blocking known bot identifiers via robots.txt does nothing by the way. Too many labs are running sneaky crawlers that do not respect robots.txt. You will need to take extreme measures: blocking basically all datacenter IP ranges, VPN IPs, aggressive rate limiting, etc. Blocking LLM crawlers has become the number one use case for our IP database customers at https://focsec.com/ |
|
|