I don't see what's so extra hoop-y about telling SSH "use key from industry standard PKCS#11". That's literally what it's there for, and when Yubico added PIV support that instantly added support to multiple operating systems (incl Linux & Windows) where SSH keys "just work".
I can take a yubikey today from my Linux system, plug it into a Windows machine, and Putty with wincrypt support "just works". Because that's how it was designed.
You're using yubikey agent or ykcs11 or yubico-piv-tool or somesuch nonsense most likely that actually provides PKCS#11. That is it wraps whatever PIV is supposed to do and gives you PKCS#11 interface. I'm talking about something that isn't Yubico specific and provides PKCS#11 and PKCS#15 with OpenSC straight ootb.
I'm using opensc straight of the box, with vanilla SSH.
I am using ssh-agent, but it works just as well without it.
I can pop my yubikey into a plain vanilla install of Linux and run "ssh -oPKCS11Provider=/path/to/opensc-pkcs11.so user@host.com".
Or just put this into your ~/.ssh/config
Host *
PKCS11Provider /path/to/opensc-pkcs11.so
Or on a Windows plain vanilla system I just pop the key in and tell putty-cryptoapi to use "the smartcard key". Windows pops up my pinentry dialog, then I touch to verify physical presence, and in I go.
I agree that the gpg-agent way is yuck. But I still don't understand what you mean about the PKCS#11 way though. Yes, setting up the key required yubico tooling. Is that what you're talking about?
If it works with OpenSC out of the box, then that's because Yubico made it to work. I pretty sure that if you tried to use a publicly available PIV applet that does its PIV duties in accordance with the specs you are not going to have a good time trying to pair it with OpenSSH.