That's like saying people shouldn't invest in farms, or pharmacies, or water pumps because there are poor or mentally disabled people who have trouble securing and affording those products.
The difference is whether the good in question is being produced by investment, or monopolized by investment. This has been enshrined in law for over a century.
You want to make more money in housing? Build more housing.
You want to make more money in housing? Build more housing.