[natrius]

If you're very interested in your question, I highly recommend Ed Glaser's Triumph of the City[1]. Short answer: Density is valuable, and developing a new dense area is difficult. No one is going to pay for a skyscraper or a cute, tiny Victorian house in the middle of nowhere because the demand isn't there. You have to build where people are, but neither San Francisco nor the peninsula suburbs want to let that happen.

[1] http://www.amazon.com/Triumph-City-Greatest-Invention-Health...

[mc32]

What about San Jose? They have lots of land and or low density tracts which they could rezone for high density -and the city is amenable to actual renewal and progress, unlike SF.

I know it has a reputation for bland mega-suburb feel (as though it were just a bunch of adjoined and conglomerated suburbs called a city) but it could change. Geology* could be an issue, if going over, say, 20 floors, but anything above 8 to 10 is good for density and could be made a viable alternative to SF hegemony in the region.

As a bonus, SJ, compared to SF has a functional gov't and has lower crime.

*http://www.sanjoseca.gov/DocumentCenter/View/2199 [PDF]