I lived until recently in a very active community of both cyclists and family-aged couples (Marin, CA, Fairfax to be specific), and heard numerous stories at the local pub about many cyclist men having this very issue. Anecdotal evidence to be sure, but enough to make me find an alternate form of exercise if/when I am trying to have a child.