It really is, in-universe. Almost noone knows this save other maiar. The wizards are perceived as men by most people, them being maiar is far from common knowledge.