I know plenty of people that live in $1MM apartments that doubled in value over the last 10 years. So while they might not be able to "spare" $50k, it's not that big a deal overall. Of course I know other people for which that would be a lifetime of savings.
Is it "silly" to buy a house when you don't have $50k in your emergency fund? I have been very financially lucky, but I'd still have to sell a lot of stock -- dipping significantly into funds earmarked for other things -- to pay off a surprise $50k expense. I certainly couldn't have done so right after having bought my house.