[Houshalter]

On the other hand the brain is insanely adaptable. People made to wear glasses that flipped their vision upside down adjusted fine, and when they took them off, everything looked upside down.

Even your example is wrong. You can in fact adjust to glasses of the wrong prescription. People who need glasses have deformed eyes which is basically like having the wrong prescription. The brain adjusts, and people can go without glasses without vomiting everywhere.

The nausea reaction is an issue, but it doesn't affect the entire population. I have friends who can't play video games because of motion sickness, but no one argues that video games can't be commercially successful. And various tricks like removing head bob, adjusting the field of view, and adding a virtual "nose", have been shown to significantly alleviate that.