[sbarre]

My specific case doesn't really matter (and my examples are from some years ago and of smaller clients, not my own setup).

My point was that people provision capacity ideally based on observed or expected traffic, and that crawlers can, and do, show up and exceed that capacity sometimes, having a negative effect on your customers' experience.

But you are correct that it's absolutely manageable. And telling crawlers to slow the F down is one of the tools you can use to manage it. :-)