It correlates, which could also explain a link in the opposite direction. Secondary sex characteristics sometimes have nothing to do with overall fitness. You can see this most clearly in animal species where the females value elaborate ornamentation, like peacocks.