How do people know that real-world QC is even possible?
I understand enough of nuclear physics and quantum physics to see that fusion is mostly a technical/engineering problem while QC is widely speculative.
The biggest reason I think its possible is that there are a lot of different ways you could build a quantum computer. Different groups are exploring building physical qubit systems out of transmons in superconductors, topological systems, linear optics, trapped ions, quantum dots, NMR driven spins, cavity QED and probably many other setups I don't know about.
It would be weird (and scientifically quite interesting) if all of these approaches fail for some reason.
You may well be correct in that bet. I lean that way myself more than most of my colleagues. On the other hand your reasoning is quite wrong, entanglement is both very well understood and fairly robust. Scale and gate fidelity are significant problems in building quantum computers, entanglement isn't really (by this I mean that building good entangling gates is hard, but the entanglement once you've made it is ok).
The theory (which is just quantum mechanics) matches the reality better than any other known physical theory ever has. It is certainly not "just literature". Of course it is wrong in some sense, since it doesn't appear to cover gravity, but it is also right in some pretty meaningful sense.
QC is just as solid of theoretical footing at fusion. It is mostly a technical/engineering/materials problem with plenty of room for clever physics to make it easier.
It would be weird (and scientifically quite interesting) if all of these approaches fail for some reason.