Only if you know for certain that the key has been accessed by a third party.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
What? This is a terrible way to reason about risks in general. If you don't know for certain, you should assume the worst case scenario, especially since it's impossible for you to calculate the probability distribution of the likelihood of a leak.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.