That is a severe misstatement of the other side of the argument. They have nothing wrong with just the act of moving, it's the act of displacing others and changing the community in such neighborhoods.
Moving somewhere always changes the community, and necessarily either involves displacement (move into someone's old apartment) or change in neighborhood character (add net new housing supply). This is a distinction without a difference.
Lots of people moving in always does one of the latter. Either they bid up the price of rent, or someone builds denser housing and thus changes the character of the neighborhood.
So it is a misrepresentation of what the other side thinks they are saying, but a correct representation of what their argument actually turns out to mean.