In short, it's both. I borrowed the idea from Bjork's New Theory of Disuse: each memory has two strengths. One decays with time, the other grows with successful recall and slows that decay.
The reranker is more of a final-stage relevance check than part of the forgetting itself.