[bot403]

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

[OliverGuy]

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

[mrweasel]

I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.