"Us" is ~50% of the population in many congressional districts. Even in the most hard left/right jurisdictions you get like an 80-20 split on federal elections. That's a 1 in 5 minority of the population. That's f-ing huge.
And between the left complaining about disenfranchisement and the right complaining about voter fraud the elected representative holds less legitimacy in the eyes of the "I didn't vote for them" minority in any given district.
Look at the crazy things totalitarian governments do to keep marginalized 10-30% religious and ethnic minorities from causing instability. A pissed off group of similar size in a western country that isn't willing to stoop that low is a civil unrest bomb waiting to go off.