Actually, I think you're really underestimating how many people struggle to find clothes that fit them well. Most people's bodies are not perfectly average. For clothes not to fit well, you don't need to be a significant outlier in one dimension. Just the accumulation of several smaller deviations from average can be enough to create an awkward fit.
Beyond that, even if we limit it to height alone, there are hundreds of millions of people who are much shorter or much taller than average.
The US Air Force learned this in the 50's. They built a cockpit based on the average body dimensions of pilots, what they ended up with is 0 pilots that were average and a cockpit that poorly fit everyone.
I can assure you having observed the process of clothes shopping for the women in my life, that as far as they are concerned, clothes do not just “fit them”.