Y
Hacker News
new
|
ask
|
show
|
jobs
by
xg15
121 days ago
Facebook just decided that instead of loading the robots.txt for every host they intend to crawl, they'll just ignore all the other robots.txt files and then access this one a million times to restore the average.
1 comments
Vinnl
121 days ago
Ah yes, robots_georg.txt.
link