There's a lot of survivorship bias in that kind of thinking. They called Galileo mad, but they also called simultaneous five dimensional time cube guy mad.
"The fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown."
- Carl Sagan