That's true, but inactive nipples don't cost anything, which certainly isn't the case for an inactive uterus. I don't know how it works, but I assume that such developments follow some kind of cost-benefit function.
afaik they serve some purpose in regulating androgenic-estrogenic hormone production.
The amount of testosterone in women is not zero, likewise the amount of estrogen in men is not zero as well, and breast tissue does serve some purpose in regulating hormoe production, even in men.
Aren't nipples pretty recent? The egg part has been there for a very long time, nipples haven't evolved as long, maybe in a few hundred million years we no longer have nipples.
Mammalian fetuses all start out the same and sexual dimorphism happens several weeks into development. The same structure that eventually develops into a uterus can instead develop into a penis/prostate. Testicles and ovaries are the same tissue early in development, just like the glans and clitoris.
Biology doesn't generally suppress one entire set of organs in favor of another. They're built from the same precursor tissue and only diverge after sex hormones are activated. Biology and evolution modify existing structures, it does not typically erase one structure and replace it with another.
In addition, intersex humans exist. There are documented incidences of males born with uteri, external genitals can form halfway between male and female. Biology can get very messy sometimes. Sex is not a hard binary switch, it's a sliding scale just like most biological features. Only most individuals are at one end or the other, there's a lot of room between.