Y
Hacker News
new
|
ask
|
show
|
jobs
by
ricardo81
2018 days ago
Hello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.
1 comments
YeGoblynQueenne
2018 days ago
Hey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!
link