And then the current residents leave before their investments (their homes) lose value and the neighborhood as a whole loses value (or turns into a slum).
It was because the new people were not white. They were black. The culture was not a factor, no matter how respectability politcs exactly as white they acted, they were black and thus a threat.