You're assuming increased height only leads to a stretch in the z axis. If you linearly scaled someone up from 5'3 to 5'4 in every direction, the increase in volume would be 5%. So maybe not the main factor, but definitely a factor.
While you have a point, weight doesn't go like height cubed (you don't grow equally in all directions). I read an article a while ago (can't find it now) that looked at the exponent for various species. I think it was typically about 2.5 rather than 3. I suppose you could look at a BMI chart and back out the exponent that is assumed.