You are confusing owning a house with having paid off a mortgage. I can go get a mortgage for $1 million tomorrow, but that does not make me a millionaire. It makes me an debtor with a house I can't afford.
You can put 3.5% down for a $1m house in places where ~50% of the population lives.
At current interest rates, no, you can't qualify, but at interest rates where people bought most of these houses... Yes, the median person could afford it (in those areas).