This seems like it could be true for memories of facts, or something you can recite, like lyrics of a song.
However for other kinds of memories, such as remembering a situation or a picture, I have heard (I'd appreciate a reference) that the recall process of long-term memory is slightly destructive, and that the memory must be re-encoded into long-term memory. That suggests that recalling a memory repeatedly could accelerate it's deterioration.
I've heard something similar, but I think it had to do with the fact that we tend to "fill in the gaps" if we can't remember something about a situation. So we just make up information. I'm not sure if there are ways to combat this by recalling the information before it starts to decay or not, but it has a lot of implications for eyewitness testimonies.
My suspicion is that we create new associations to the old memory. For distinctly visual memories ... I'm not sure what the mechanism is, but at least for myself, I often don't recall specifically what something (or someone) looks like, until I see them again. The act of recall-without-seeing may overwrite the actual recollection with what you think it should be.
I don't know about the brain acting like a cache, but synaptic pathway weighting is thought to be modified by how often neurons on either side of it fire near the same time. This is called spike timing dependent plasticity (STDP). The seminal paper for this is: http://www.jneurosci.org/content/18/24/10464.long
It's an analogy for the reinforcing characteristic of neuronal connections, would be perfectly adequate if not for the continuum vs discrete difference in the reinforcement mechanism.
However for other kinds of memories, such as remembering a situation or a picture, I have heard (I'd appreciate a reference) that the recall process of long-term memory is slightly destructive, and that the memory must be re-encoded into long-term memory. That suggests that recalling a memory repeatedly could accelerate it's deterioration.