The overall portion of athletes are vegetarian or vegan is low, but the ones who are can still perform very well. This suggests that the difference is based, at least in part, on preference.
There have also been many athletes who tried vegan diets and found that their performance declined, or they had more difficulty recovering from injuries. So preference is only one factor.