Makes sense except for one point: Why would a person purposefully subject themselves to a musical genre that sounds bad? You said yourself, to the virgin ear, jazz sounds bad. That means that everyone who likes jazz music at one point thought that it was terrible music yet listened to it anyway.
You are not used to listen to Jazz so it's more likely to sound horrible to you since it's often rather complex.
Why do you think that parents of the 68' generation hated Beatles despite it being what we consider pop culture today.
You need to understand this in context rather than in absolute terms.