I understand what you are saying but in my experience the opposite can be equally true. Unless every system in your org is managing the authorized_keys for every account with automation and hourly validation one can end up with rogue keys, forgotten keys, unmanaged keys. SSH has no concept of identity in this regard. If I temporarily have sudo on a system I can append any random public key into the authorized_keys of any account. It gets even worse if that system allows passwordless sudo. Now there is an audit trail that showed you made a reckless change when it was really me. This risk gets exponentially worse if the system I append a random key into is a command and control or configuration management system. One doesn't even really need sudo for this to be a problem given how much power non root accounts have over applications, deployments, builds and secret management.
Is authorized keys important here? If I send the server my password, it has my password. If I send a server my public key, that's a lot less severe. If an attacker is on the server I expect that the difference is obvious?
Both are vulnerable to phishing via malicious scripts. One could even argue that keys are more dangerous if the attacker can append their own key into authorized_keys one host at a time. Some might be surprised how incredibly easy it is to get a percentage of technical people on an email distribution to execute a script without fully understanding it and this is even leaving things out like compromising npm repositories. Said script can drop a key into authorized_keys or drop a custom key and create an outbound session invoking a gateway port allowing the attack to ssh to the host even if they have no inbound ports open. From there one can repeat the process as access is gained first at the workstation or laptop, then the jump host then build servers, repository servers, staging servers and ultimately production servers.
SSH is a power communications tool. It is equally powerful to both the authorized user and to attackers in my experience. It is often poorly managed if even managed at all even in the most sensitive environments in some of the most high profile organizations.
I would not. I would phish you running a script that adds my public key to your authorized_keys and would ssh from your client in the background creating a gateway port back to you. At that point I can get your ssh keys and just about anything else. If systems in your org allow ssh multiplexing default is enabled then I can also piggy-back your sessions bypassing MFA/2FA.
I say you but I mean most people. I'm certain that you specifically would not fall for it but about 10% [1] of technical people will.
[1] - stats based on past career and testing of a large technical population.
I'm not sure I understand the attack. I connect to a server that you, the attacker, are on. How does your attack get my private key? Assuming I'm not forwarding my key, and you're not on my laptop.