Maybe because it's an overly simplistic LRU cache, in which case a different eviction algorithm would be better.
It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.
It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.