They're not all taking banned substances. Case in point, Hergie Bacyadan and Elis Lundholm competed in the last Summer and Winter Olympics respectively.
Both Hergie Bacyadan and Elis Lundholm has not undergone any hormone replacement therapy or surgery, and competes in the women's divisions. Their status as trans men has nothing to do with their eligibility to participate.
This would be like if two trans women, who has not undergone any hormone replacement therapy or surgery, would compete in men's divisions.
This would be like if two trans women, who has not undergone any hormone replacement therapy or surgery, would compete in men's divisions.