That's true only if the selection pressure on individual children favors largeness, right? On an island or without as much nutrition, it might be an advantage to be smaller, since you could get by on less food.
That's a prisoner's dilemma situation. As a group, smaller islanders are less likely to run out of food (the cooperation strategy). But as an individual, a bigger islander can beat up the smaller ones and take their food (the defection strategy).
Of course if everyone defects, the group as a whole is worse off.
I dunno. If they really were trapped in a Prisoner's Dilemma, then the equilibrium would be that they get big, I think (unless there's a "repeated games" angle here). Yet what we actually observe, of species that get isolated on islands, is that they do get small.
Of course if everyone defects, the group as a whole is worse off.