How did you arrive at that conclusion? The only studies I could find show no such correlation[0][1].
I remember reading that vegans had higher levels of testosterone when not controlling for body weight, as vegans tend to be less obese than non-vegans. Can't find the source, so pinch of salt here. Makes sense though.