Now let's take the other extreme. You split the minority population among 10 different seats where their population in each district is so small that their interests are ignored entirely in each one.
That's what coalition-building (an important feature of other governmental structures used outside the US) comes in. You generally have a bunch of small minority groups all over the place, but if they work together ("hey, we'll support your group's big issue if you support ours"), they can form a large enough voting bloc that their issues get traction.
If your population is relatively balanced then you need to worry about everybody because every vote might count. Sure a minority might only be 5% of the population, but if you anger them and they vote for your opponent that could absolutely change the outcome of an otherwise close race.