[blinkedup]

The other men he competes with have similar physical advantages. And elite female swimmers do too.

But it would be completely unfair to have Phelps compete in the women's events, because he also has the male physical advantage, which female swimmers do not have.

Which we can see most starkly by comparing the record times of equivalent female and male events. The best of the best male swimmers are considerably faster than the best of the best female swimmers.