[tialaramex]

But you've now swapped out part of the authentication. If you want to claim that, for some threat models, "One password per server" is better than "One key" then, sure, so use "One key per server" and now keys are better again.

Also - I suspect this Caddy server doesn't support it, but OpenSSH does - you can use FIDO and then the keys physically are objects in the real world, from say Yubico or a dozen other vendors so now "losing the keys" is like losing your office keys, except that when they give you a new one they can trivially make the old one stop working.

[getcrunk]

A users keys are unlikely to be compromised singly. You would get them all. Whatever malware exfils the keys would take all that it finds.