There is an ideal size and it’s determined by the size of the island. For a given size of island there’s a maximum size of predators that it can support. You want to be big enough that these predators aren’t much of a threat, but small enough that you can keep your own population size viable.
I wonder why humans don’t exhibit this. Is it because we can come up with ingenious ways to feed ourselves and aren’t completely dependent on the environment? Or maybe we do, south East Asians are really small compared to say a Polynesian.
There was Homo floresiensis, a relatively close relative of ours, also called the "Hobbit". For modern humans, there simply are no known populations that have been isolated for long enough on a small enough island to exhibit dwarfism. We're pretty good with boats and we supposedly have been very mobile for most of our history, and even if stuck we're capable of controlling our numbers and exploit our environments very well. I believe there are adaptations among Pacific Islander communities that increase metabolism efficiency and similar traits so we're not immune to this kind of pressure.
I wonder if it is just a matter of timescales? Behaviorally modern human showed up, banged some rocks together, and then launched satellites into space ensuring that we’d never truly be isolated in an evolutionary blink of the eye.
What’s even worse is that Island Gigantism is a Germanic derived word followed by a Latin derived word, while Insular Dwarfism has the etymologies the other way around.
Reversion to the mean is a phenomenon resulting from random sampling. In island gigantism/dwarfism, the distribution of the population changes due to various causes. So not really the same at all, as far as I can see.