The danger here is that anyone with physical access can grab the keys used for the SSH server and set up a man-in-the-middle attack to grab the encryption passphrase.