Yes. There as difference between unfair and unreal; someone who is malnourished when growing up will forever likely be weaker than someone who received a proper sequence of meals.
We should perhaps recognize that and try to compensate for it, and it's not a value judgement on the person so afflicted, but pretending it doesn't exist just confuses matters.
That's been the entire fight over the last 20+ years, does the test identify anything real and if so, what should be done with it (equality of outcomes vs equality of opportunity, e.g.).