I use my yubikey and I love it. I have it set up to do GPG, SSH, TOTP, and U2F and it works great. It is worlds better then any other Smart Card or second factor out there, and U2F is literally just plug it in and tap it.
Have you got a writeup of the ssh setup methodology you used?
(I've tried scouting around, but not found anything clear yet. Someone's done native support in ssh, but the patch set is hung up on licensing issues and technical quibbles[1], and some of the PAM-based setups seem to require cut-and-paste of crypto strings on every login.)
But my security model does not allow putting myself in a position where I am stranded without my second factor (or doing huge amounts of work re-registering everything).
The problem is largely with their docs, or lack thereof. Just figuring out how to use one as a token for ssh is incredibly painful. The docs are very "enterprise," meaning half-done, overly complicated, confusing, scattered, etc.
There was a series of changes, e.g. regarding platform keys, software source availability etc etc. I think there were some "I don't endorse this anymore" posts, although I don't really remember the details.