Hacker News new | ask | show | jobs
by SageRaven 4869 days ago
Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?
1 comments
kanzure 4869 days ago
The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.
link
dsl 4869 days ago
500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.
link