[wyldfire]

> There is also a generous crawl delay between pages on the same host.

What's the order of magnitude of this delay? milliseconds? hundreds of milliseconds? seconds? I'm curious what's considered 'polite' in this realm and how the various parties come to form opinions on this.

[ricardo81]

A minimum of 4 seconds.

[YeGoblynQueenne]

I just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page visits:

  User-agent: bingbot
  Allow : /
  Crawl-delay: 10

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...

[ricardo81]

Hello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.

[YeGoblynQueenne]

Hey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!