Y
Hacker News
new
|
ask
|
show
|
jobs
by
kanzure
4869 days ago
The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.
1 comments
dsl
4869 days ago
500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.
link