OK, that's a rough argument that needs fleshing out. Has the rigorous argument actually been made and is widely accepted by economists or is this just a meme that's accepted and repeated without much basis?
Adam Smith makes the argument well in A Wealth of Nations. Essentially, the only reason to invest is that you expect to earn something above on top of the risk. Then add that up over everybody in the system. They cannot all get something extra, unless there is growth. That growth is spread over everybody as return on investment. So, if there is no growth, then investment stops earning that extra, so there no point in investment, so nobody hires anybody or builds new machines.
Importantly, it's growth in total value, not growth in number of things. Hence, it's okay that growth is infinite without gobbling up the universe - we can always be better. Stuff can always be sexier.
Same goes for food, etc. my point was that not everything has to be for financial gain -- we have basic needs that are higher in the priority stack and will always be there.
About homes specifically - they have a life span too so they eventually have to be rebuilt or, as others have pointed out, maintained.
There has got to be some ways in which this basic economic assumption is wrong. Like with cars, I can buy a car and when I sell the car it will be worth much less. But I will buy that car anyways because I need a car.
Or for a business perspective, I buy a dairy cow as an investment, when I sell the cow it will be worth much less than what I bought it for, but I don't mind because the cow has provided a steady stream of milk based income over the years.
So in these ways at least, it doesn't seem like economic prosperity is always dependent on growth.
Maybe in some economic circumstances cash might be a better investment or worth more than my dairy cow, but such a circumstance can't last for long because supply and demand affect money too right - so if enough people get rid of their dairy cow for the cash, eventually that would decrease the supply of milk, increasing price which would put me and my milk business back on top again would it not?
But surely people invest for "maintenance" purposes too, where nothing extra is expected. I invest in maintaining my house. I don't expect any "return" from that investment, besides the roof not collapsing.
Maintenance is obviously limited to the number of things maintained and cannot stop deflation, because costs will be approaching zero due to innovation and competition.
Edit: And you do expect a return. You maintain your house, because it has a value and rent costs money, so you have more house in the future if you maintain it now. Imagine a world were rent and the value of your house is approaching zero, would you maintain it? You would have more house if you just kept your money and rented a house.
Because you liquidate all your assets if there is deflation. No companies, no jobs, no food.
Edit: Obviously I am speaking in very extreme terms and ignoring the fact that there is a government acting and that not everything will happen instantly. These are just long-term forces.
The problem with this argument is that it mixes global effects and personal effects.
A person could have an appealing investment - for example building a more attractive fruit shop, with better customer service - taking all the sales from the current fruit shop .
Here's an interesting theoretical argument that growth can't continue forever (not saying this applies now):
"[I]f energy became arbitrarily cheap, someone could buy all of it, and suddenly the activities that comprise the economy would grind to a halt. Food would stop arriving at the plate without energy for purchase, so people would pay attention to this. Someone would be willing to pay more for it. Everyone would. There will be a floor to how low energy prices can go as a fraction of GDP."
Well, it does. If there is a limit on how small X can be as a percentage of the total economy (because we value it), and the amount of X that can be produced is finite, this puts a limit on the size of the total economy.
Importantly, it's growth in total value, not growth in number of things. Hence, it's okay that growth is infinite without gobbling up the universe - we can always be better. Stuff can always be sexier.