You should only rely on your memory for passwords that you use frequently. Rarely-used passwords should be kept somewhere safe and well-maintained, such as a password manager and/or on paper.
Rarely used passwords is often a "reset password" anyway. But if it's some store or whatever that I use maybe once every 5 years, does it matter what the password is? My point was this: for most accounts, there is no risk involved with anyone guessing my password. It doesn't matter whether I return to a store and some hacker has guessed the password or they were poorly hashed and their database leaked and my 12345 password was swiftly reversed. Because all you can do with my password on that store is... I'm not even sure what it is. Post spam in product review pages?