Moving to a different town is a big disruption which at least i won't do just to be able to afford a big house. Not only do you lose the option to come in to the office (which is a good option to have even if you prefer home office most of the time); some people just prefer to live in big cities.