Hacker News new | ask | show | jobs
by everfrustrated 22 days ago
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
2 comments

Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
As someone who operates a wiki, this does not solve the problem.