Maybe I misunderstood this point. But the ssh socket also gives access to your private keys, so I see no security gain in that point. Better to have a password protected key.
It's so your private key is not stolen, but you're right passphrase protected keys win anyway. I use hardware keys so this isn't a problem for me to begin with.