[dkl]

He's why it's not silly and why disabling password logins and using key-based logins in SSH doesn't increase your security:

If you turn off password logins, people will use authorized_keys to in effect get a password-less login. If their public key has a password, this is OK, since they're either using ssh-agent or typing in their password at the time of the login. However, what if their ssh key has no password on it? That gives a password-less login path to my host, which is less secure. The problem is, it is impossible to detect, on the server side, a login with a key with no password.

[tptacek]

I'm failing to see what this has to do with the threat model that port knocking addresses. Suffice it to say that an attacker who has access to your SSH identity file has with virtually total certainty access to your SSH passphrase as well.

[dkl]

I'm failing to see what this has to do with the threat model that port knocking addresses.

You said:

I think port knocking is really silly and you shouldn't waste time with it. Disable root logins and password logins in SSH.

I'm pointing out that will lead to less security, not more, for the stated reason.

Suffice it to say that an attacker who has access to your SSH identity file has with virtually total certainty access to your SSH passphrase as well.

How so? By brute for cracking it?