Your median sale price index compares house prices in some random Southern hell hole in Mississippi to a disaster ravaged area of Puerto Rico with NY and SF or Atlanta and Austin house prices. Places where people can actually go to work.
The middle can shift to the right easily if you exclude houses in some random houses in the middle of nowhere, while retaining exclusive country neighborhoods that people still move to and commute to and from for work.
The nice thing about the median is that it's robust against outliers like the ones you've mentioned. It's over $300k, so we're definitely not talking about some hellhole in MS.
How about median house prices in all major employment zones in the US? I'm sure that would filter out all the small places in the country while also including places in the country used by the old money Brahmans. In either case, you still can't compare house prices in Albany to house prices in NY, which these median measures happen to weight in, even if slightly. Exclude a thousand such places and the median shifts to the right.