What would keep a side-channel attacker from reading your YubiKey PIN from kernel memory, and then racing all your authentication attempts until it manages to win the physical button event?
Nothing maybe. But if a side-channel attack gains access to kernel memory, they can do just about anything, and no login scheme is secure against that.