Three kids here. I could never smell it either. To me babies mostly smell of poop and whatever medication of the hour you have slapped all over their bottom.
One thing I have noticed is they smell worse as they get older :)
You remind me of my sister. Once her son was a tween, she remarked "omg, <son's name> has reached puberty. I have to hold my breath to collect the laundry."