[jdcarr]

The important thing you need to assess is how critical is it that clients receive fresh data.

You can imagine that for a real time service it would be better to provide a timeout immediately rather than providing stale data.

HN is an example of a near on-line site where some delay is perfectly acceptable. No one cares that they're receiving a 2 second old page, it's better for the site users to reveive old data fast rather than new data slow.

If you use nginx the following commands would help out significantly (if I remember them correctly)

proxy_cache_use_stale updating proxy_cache_lock on proxy_cache_lock_timeout 1s

This config allows nginx to fetch cache updates while serving clients and when fresh data is received from the upstream application server it'll use that immediately.

If that's wrong hopefully someone can correct the conf.