Again, you can be sure, by comparing the safety numbers. It's the same as comparing SSH or GPG key fingerprints. If someone else masquerades as Bob, the numbers won't match. See section III-D3, key fingerprint verification:
https://www.ieee-security.org/TC/SP2015/papers-archived/6949...
That would be true if people routinely verified fingerprints of their contacts. I don't think it happens more often than any other commonly ignored security precautions. Also, what happens if the phone is lost/damaged/replaced with a newer model? I assume new key and thus new fingerprint?
Signal, WhatsApp, Matrix et al. show notifications when the participant’s device keys change. You’re right that most users don’t verify. The opportunity for detection or prevention of common forms of surveillance is better than none at all.