Makes sense except for one point: Why would a person purposefully subject themselves to a musical genre that sounds bad? You said yourself, to the virgin ear, jazz sounds bad. That means that everyone who likes jazz music at one point thought that it was terrible music yet listened to it anyway.
According to this website I listen to 60 tracks every day.