[mmaunder]

PG: welcome to the woes of being an alexa top 1000 site with over 1 million pages of dynamic content.

HN has roughly 1.3 million pages indexed by google.

1.3M pages at 43k per page is 53 gigs to cache static versions of all pages on the site. Quadruple that for a worst case scenario and it'll still easily fit on a single drive.

When your site gets this popular you tend to have to re-architect your application to solve perf issues. You could serve googlebot UA's 1 week old cached pages for example.

I'd encourage you to start thinking of yourself as a utility providing a valuable and necessary resource to the Net and take the time and energy to solve this properly.

[resnamen]

I'd like him to fix the experience for non-bot users first, namely the "Unknown or expired link" issue.