Software performance, which translates into CPU/memory/disk resources, are only one aspect of the costs incurred by crawling bots, and fall under the unmetered / virtually infinite category. However, there are also the metered resources, that do translate (after a certain threshold) into monetary costs: network bandwidth.
Thus, regardless how well one optimizes his site delivery (static site, minimizing, CDN, caching, etc.) a stampede of bot crawling does in the end become a DDoS, which if it doesn't take down the infrastructure, it might leave a deep hole in one's budget.
For example, for one of the sites I manage, I get daily peaks of ~300 requests per second measured at the backend, for a site that already employs heavy caching (both client-side, CDN, and even server-side). This wasn't so a few months back, and the site didn't just jump in popularity.
Thus, regardless how well one optimizes his site delivery (static site, minimizing, CDN, caching, etc.) a stampede of bot crawling does in the end become a DDoS, which if it doesn't take down the infrastructure, it might leave a deep hole in one's budget.
For example, for one of the sites I manage, I get daily peaks of ~300 requests per second measured at the backend, for a site that already employs heavy caching (both client-side, CDN, and even server-side). This wasn't so a few months back, and the site didn't just jump in popularity.