|
|
|
|
|
|
by nostrademons
2617 days ago
|
|
|
Because then you end up in an arms race that the little guy usually does not win. There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations). |
|
|