The risk of genetic defects increase with age[1]. You are right though, you can definitely have healthy children in your 30's, though late 40s looks a little scary.
The pattern I have seen is enjoying your 20s, then deciding to have a family, but not having a suitable partner, which leads to undesirable outcomes (desperate decisions, never finding a partner, disappointment of not engine up with kids, difficulty conceiving). This goes for men and women.
Another pattern is seeing people in their thirties finding it harder to adjust to the necessary changes because they have got accustomed to a lifestyle.
Don't know you're grayed out. Humans are genetically extremely similar, appearances not-withstanding. Nurture is what creates valuable and functional humans. Societal factors that reduce parents ability to nurture their children are bad regardless of the parents gender.
Trying to set up the perfect nest, or to save "enough", is not a game that most people can win.
At least, the above is from what I have seen of friends that had kids, compared with those that delayed too long.