Yes, it's risky to accept password auth if someone sharing the box with you has a poor password. They could do things like:
. Install a spam or brute force password bot, which could get the machine kicked off its internet connection (in addition to whatever havoc it causes first)
. DoS the server by filling up the disk or using too much RAM (are quotas enforced?)
. Exploit a local vuln to get root, if such exists on that box. (Is the kernel promptly patched and the box rebooted?)
. Explore other users' directories (are permissions locked down correctly across users?)
It is, if for no other reason than you never know when some other user has a guessable password. You should switch everyone to ssh keys. It's a good excuse to learn :)
How good is your password? If it's long, with special characters, it's fine. Install fail2ban. The problem with auth keys is you can't get into the server if you don't have your laptop/phone/NFC device because you got pickpocketed/mugged?
. Install a spam or brute force password bot, which could get the machine kicked off its internet connection (in addition to whatever havoc it causes first)
. DoS the server by filling up the disk or using too much RAM (are quotas enforced?)
. Exploit a local vuln to get root, if such exists on that box. (Is the kernel promptly patched and the box rebooted?)
. Explore other users' directories (are permissions locked down correctly across users?)
…and more thrilling possibilities!
Embrace key auth. Future you will thank you.