Agreed, there are probably some stability concerns. But I expect the largest aspect is just energy usage. Brains use a lot of energy and until recently it was the primary limiting factor.
Otherwise we would also all have more muscle mass. There doesn't seem to be much downside to that.
Humans have (IIRC nearly unique) ability to have adaptive muscle mass. Gorillas get huge muscles without needing to exercise and by only eating vegetables, but we grow and shrink with our diets.
This is helpful because it means we can survive with unpredictable diets. But there might be other benefits like heart health or avoiding cancer.
Another big issue with huge brains is that your babies take longer to develop and are very hard to give birth to.
And your babies have to be born exceptionally helpless and vulnerable, often taking a year or more just to walk, whereas many animals begin walking in a matter of minutes or hours.
A significant portion of their development has to be moved outside the safety of the womb or their heads would get them stuck!
> or to be tantamount to training on IQ tests themselves which destroys their meaning (like memorizing vocabulary)
I have trouble understanding how people can make this point and still be big believers in (utility of?) IQs. If training for IQ tests destroys their validity they never had much in the first place.
Otherwise we would also all have more muscle mass. There doesn't seem to be much downside to that.