That can’t be the reason. If you _change_ the password (and use good passwords that are not correlated with each other), an attacker learns nothing from being able to read your old password.
The key is also stored on the disk, encrypted with the password. So presumably an attacker reading the disk could recover the encrypted key and the password and use those to recover the encryption key, which they could then use to decrypt any un-erased fragments of files they find on the disk.
Or something like that. It's an admittedly implausible scenario, but good security is not based on "meh, it probably won't happen". Of course, a much more plausible scenario is password re-use.