I think the truly toxic thing is people not spending money in their communities. One of the things I pick up from anti-gentrification stories is that when new people move in, the existing shops lose customers. No-one goes to the corner store, they get food via an app or wait for a Whole Foods to open.
The (completely unintended) effect of this is that the existing community gets poorer, not richer, when rich people move in.