[griffzhowl]

If someone steals my passwords and then does nothing with them, or just uses them for their private purposes, then there's no problem. The problems only occur if my passwords are used to take control of my accounts or identity, which would deprive me of my accounts or money etc. So your example actually reinforces that the relevant ethical distinction (the harm) is indeed in intending to deprive someone of something they possess/control

[TZubiri]

I don't think this is the case legally, it might depend on the facts, but usually passwords are stored on your systems, and an attacker would have to not only access your system, but to exfiltrate that data.

It would constitute computer fraud and abuse by most definitions. This is relevant because it is sufficient to prove someone has your passwords in order to convict them, you don't need to prove they used them maliciously. (Provided of course they are a third party with no legitimate reason to have your passwords)