That's true. There is more intra-group variation because within any group you have geniuses and you have morons, which is the widest spread you can have either intra-group or inter-group.
If every group is a bell curve, then the curves have similar variance and similar mean as well, and the variance within each group is larger than the difference between the means, and the differences are not big enough to say in a meaningful way that one group is smarter than another.