The problem is that moving is economically inefficient and a large part of the population will be priced out and have to continually move so you need some hysteresis.
Moving is extremely economically efficient. Older people should not live close to the financial district, people with kids should cluster around schools and playgrounds, people without cars should live closer to transit, etc, etc
Sorry is "real estate agents need their cut" an argument against moving, or an argument for reducing the power of real estate agents, who are pretty much useless for 99% of residential transactions anyway?