It surely helped to change how black people are treated in America, no government-mandate would mean "no blacks allowed" signs still hanging somewhere to this day...
Yet we don't allow random traveler to vote in local elections.
Modern travel is big specifically because the world is diverse place. If all societies would try to include anyone from anywhere, they'd become similar and traveling wouldn't be as great.
By that logic you'd have to argue that children, inmates, mentally ill and those that for whatever reason didn't register to vote in states where it's required are not the part of society.
Including outsiders in the society doesn't make societies less diverse. If anything it's making them more diverse because otherwise similar societies might include different outsiders and differ more because of that.