Aside from situations where high security is needed (which would necessitate a dedicated piece of hardware to be the 2nd factor), Why are you adverse to using a smartphone as the 2nd factor?
I don't own a smartphone, for one, but I could purchase one if it would benefit me. As it is though, I cannot justify the cost and the increased monthly expenses. But for the sake of argument; if I did own a smartphone, I wouldn't want to turn it into a sort of virtual keychain. I can buy three U2F keys for under €60, and have spares. Smartphones are a single point of failure — unless you are in the habit of collecting spare smartphones.
There is also the matter of control. At the moment there are only a handful of smartphones that run an OS that is not under the control of Apple, Microsoft, or Google (Ubuntu smartphones come to mind as an alternative). On the other hand, on a normal computer I can run a full free software stack. To me this seems odd. If I own a piece of hardware, I want to run an OS and applications I can legally modify and tinker with, especially if I use it as a security device. By using tools such as Google Authenticator I get sucked into this walled garden.
The behaviour of hardware tokens such as Yubico's U2F keys on the other hand is completely verifiable — mostly because they are quite simple compared to a smartphone.
(Aside from these personal views on software, I do have high security requirements as well for some services, where a dedicated hardware token is preferable, as you point out.)
Hardware tokens are general purpose computers that have been locked down to the vendor's walled gardens. Yubikey is a JavaCard (in USB stick form factor) running some closed-source applets. There is a key that would allow you to load your own applets onto it but Yubico won't tell you what it is.
From a software freedom perspective, hardware tokens/smart cards are no better than smartphones.
There is a crucial difference here. With FIDO U2F I am not limited to Yubico's offerings; there are other vendors. FIDO U2F devices can be manufactured by any company (and are).
I agree that the ideal situation would be a device that is completely under the control of the user, but compared to IOS or Android, a hardware token that only does what it is supposed to do (in the case of the U2F tokens; register and sign authentication requests), and cannot be modified by the manufacturer after sale is very much preferable.
That's not a difference. You are also free to choose among many different smartphone manufacturers and OSes, all of which are capable of running a TOTP app.
I had the same question as you and the only thing I could think of was that our phones can be hacked too. It's a publicly facing device as well and has multiple vectors for attack vs. a standard dedicated 2F auth device.
The problem also is that the smartphone can be hacked and opens a huge attack vector that you don't have with other systems.
Also, typing in stuff by hand is a general bother.