[solid_fuel]

Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.