As a thought experiment, imagine that you could rent a million dollar house for 1$/year. Then renting is obviously the smart thing to do. So rental yields are an important factor, as are your expectations of house price growth, opportunity costs etc.
But if rental yields are low (as they are now) and you do not expect house prices to rise at a fast rate in the future, renting can easily be the smart choice.
Regarding future house price increases. It seems to me that house prices can be approximated as a function of rents, interest rates (determining required return on owning a house), and anticipated growth in house prices (which again can be decomposed into growth in rents, reduction in interest rates and 'irrational' growth).
Using London as an example. Rental yields appear to be around 3.5% in London at the moment, which suggests to me that a good deal of anticipated growth is priced in. But where can that growth come from? Rents are a function of incomes and are unlikely to outpace inflation. Interest rates seem to be more likely to go up than down. So it seems that house prices are unlikely to rise, and once that is realized by the market, prices may even come down as participants stop anticipating growth just because "London house prices always go up".
House prices going up faster than rents in recent decades can probably be explained by interest rates being in a downward trend over the same decades. Now we seem to be in at least somewhat of a bubble due to irrational future growth expectations.
I'm general, when the price-to-rent ratio is too high. Last I heard San Francisco averages >30. It only makes sense (financially) to own if the price-to-rent ratio is under ~15. If you're an investor looking to become a landlord don't even consider buying a property if the price-to-rent ratio is greater than 8.
Its a smart choice in other situations too - like when you are intending to live somewhere on a temporary basis, for example. You would certainly be better off (again, financially!!) if you were renting a room for a couple hundred bucks a month if that allowed you to sock away $2,000 a month into your retirement funds. That's provided you have the ability to actually save/invest the extra money you have, a mortgage can act like a forced savings account for some people.
Its too late to edit this post, but I should be more clear that the parent post is a major generalization, your individual factors will vary if it makes sense (financially!) to rent vs buy for your primary residence. Price-to-rent ratio is a big factor, but not the only one. There's also a lot of luck and factors out of your control that contribute to if you make out ahead (financially!) either way. I just looked up the data on Zillow and the average P/R ratio in San Fran isn't as bad as I thought, but still extremely high.
Anyway, when you're talking about buying vs renting you're also heavily involving non-financial factors that are usually MUCH more important than financial factors (assuming you have some sort of a budget); so something might be a "smart choice" for an individual but not the most financially optimal one. Just like eating lentils and ramen noodles every day might be the most financially optimal choice but is probably not a "smart choice" for most people. In fact, if you have a family you probably aren't going to make the most financially optimal housing choice unless you absolutely have to.
A real world example:
I have friends who are renting a decent single family house with tons of land for an absurdly low price (family price!) and the landlord (family member) even pays utilities because it's just "easier" and she isn't trying to make a profit. Factoring in depreciation and amortized repair/upkeep costs the landlord is almost certainly subsidizing their housing costs. Financially, great deal! They will almost certainly never have a chance to have a full house that cheap ever again. Staying there as long as the landlord lets them is the most financially optimal choice. However, for a number of complicated (but understandable) reasons, my friends hate it and are planning to move whenever they have saved up more and paid down their loan balances.
There's a great writeup on this -- I may have even seen it here [1]. On average house prices tend to keep up with inflation and trail the market across the US. In specific markets that may not be true, then there's all your expenses. Ultimately, it depends on your personal situation if you should buy or rent, and the New York Times has a great calculator [2].
tl;dr: It's not always the smart choice to buy. Or to rent.
But if rental yields are low (as they are now) and you do not expect house prices to rise at a fast rate in the future, renting can easily be the smart choice.
Regarding future house price increases. It seems to me that house prices can be approximated as a function of rents, interest rates (determining required return on owning a house), and anticipated growth in house prices (which again can be decomposed into growth in rents, reduction in interest rates and 'irrational' growth). Using London as an example. Rental yields appear to be around 3.5% in London at the moment, which suggests to me that a good deal of anticipated growth is priced in. But where can that growth come from? Rents are a function of incomes and are unlikely to outpace inflation. Interest rates seem to be more likely to go up than down. So it seems that house prices are unlikely to rise, and once that is realized by the market, prices may even come down as participants stop anticipating growth just because "London house prices always go up".
House prices going up faster than rents in recent decades can probably be explained by interest rates being in a downward trend over the same decades. Now we seem to be in at least somewhat of a bubble due to irrational future growth expectations.