I've only used a debugger a few times, when the program did not seem to behave according to the source code. That was invariably due to a third-party bug (compiler, library, OS) or to me failing to understand a subtle point about the language or library I was using.
I don't think this is a right analogy, because the code can't "go bad" on its own one day (as if some capacitor would go dry), unless you modify it to do so. Maybe when you don't have access to the source code and you want to see what is going on, use of debugger could make sense.
But in general I agree with you.