There is no evidence that saturated fat inhibits athletic performance. Vegan bodybuilders are a small fraction of the total, and that's just one sport. What percentage of elite athletes across all sports are vegan?
The overall portion of athletes are vegetarian or vegan is low, but the ones who are can still perform very well. This suggests that the difference is based, at least in part, on preference.
There have also been many athletes who tried vegan diets and found that their performance declined, or they had more difficulty recovering from injuries. So preference is only one factor.