[ingenthr]

A couple of points. Some deployments do query from replicas already, but we can make this a bit better in the vbucket world. A second point is that moxi actually has functionality to non-coherently cache some items locally. Moving the cache closer to the application is always the best thing to do if you're willing to relax consistency. This is better than having it elsewhere in the cluster. The fastest network read is the one you don't actually have to do.

moxi does this with a key prefix and a timeout per item (even 1 second makes a big difference for hot data), then fetching it from the authoritative server.

We've not really exploited this moxi functionality in membase deployments yet, but it's been there in moxi for a year now and could be easily turned on if we had a deployment which needed it.

Having said all of that, with membase (as with memcached) the limit you'll hit first for an item you're fetching is network throughput. It's not that common to be fetching a few items so much that you overrun network. Even if you do, then the answer is simple: add a few more nodes.