Why is SF (I assume you mean local government?) anti-height? Are there downsides to building upwards (environmental? taxation?) or is it just a refusal to allow the landscape to change?
It's fairly common European cities to be anti-height in the city center for reasons that seem somewhat similar to SF's, though seems less common elsewhere. Some of it is a reaction (sometimes overreaction) to tall modernist housing blocks which nobody really liked. That and other things led to a preference for "human-scale" 5-7-story mid-rises. Another factor is a desire for light to be able to hit streets and parks in the "common areas" in the city center. A common compromise is to have a mid-rise historical center, but high-rises a bit outside of it, connected to good transit, e.g. central Paris banned skyscrapers after the Tour Montparnasse was so badly received, but there are skyscrapers a short distance away in the "new downtown" area of La Défense. An SF version of that might be to keep central SF mid-rise, but allow high-rise towers near the BART/Caltrain stations in Oakland, Daly City, and Burlingame.
The Bay Area's problem is that nobody wants the high-rises: putting them in the center or putting them near transit a bit outside the center would both work, but everyone, except to some extent Oakland, is anti-development, so they go nowhere. Heck, Palo Alto won't even allow mid-rise apartments near the Palo Alto Caltrain, so you actually have people "reverse commuting" from SF down to Stanford, because as crazy as living in SF is, trying to live in downtown Palo Alto is even crazier. A few livable, urbanized downtowns on the peninsula might siphon off some of the demand in SF, not solving the problem, but at least reducing it.
Other problems that seem like they exacerbate SF's problems: 1) poor intracity transit, e.g. lack of an east-west subway, concentrates demand in a handful of eastern districts; and 2) large areas of the city are not even mid-rise, but full of two-story houses (partly historical, partly anti-development, partly related to #1).
"f you look at San Francisco’s zoning map, you’ll see height and density restrictions everywhere. There are also citywide building caps — restrictions on the number of new buildings that can be started every year. And finally there are labyrinthine regulatory procedures. Here, for your amusement, is a flowchart put together by the San Francisco Planning Department that outlines the steps a developer needs to go through to obtain a building permit. Despite the department’s use of Comic Sans, this is not meant to be a joke."
So the government passes laws to limit housing stock, then passes laws to artificially keep rent low, then people complain about prices going up and a lack of housing stock? Has anyone in SF read an Economics 101 textbook?
I'm sure they understand that, but they understand 'elections' and their constituencies even better.
Allowing building upwards would be seen as 'pandering' to 'special interests' by lots of locals --specially people who like to call themselves Xth-generation Franciscan.
SF is always about the short-term but made to look long-term so as to seem forward looking and progressive. The only thing SF is progressive about two things, sex, and some minority causes (both of which are good things but hardly encompassing of the term 'progressive'.)
SoMa is the only neighborhood with any 20+ story residential buildings going up. It's hard to say why so many of the other neighborhoods are against building up. The geography of SF lends to some beautiful views from atop the hills. Residents living on those hills are unlikely to welcome tall buildings to crop up and block their view.
SoMa probably ends up being the big-city, tall building part of SF in the future. It would take a big mind-set change for many other neighborhoods to build up anytime soon.
There are some city blocks in the Financial District where you're perpetually in the dark and are essentially in a wind tunnel due to the tall buildings. It's creepy as hell. :)
The Bay Area's problem is that nobody wants the high-rises: putting them in the center or putting them near transit a bit outside the center would both work, but everyone, except to some extent Oakland, is anti-development, so they go nowhere. Heck, Palo Alto won't even allow mid-rise apartments near the Palo Alto Caltrain, so you actually have people "reverse commuting" from SF down to Stanford, because as crazy as living in SF is, trying to live in downtown Palo Alto is even crazier. A few livable, urbanized downtowns on the peninsula might siphon off some of the demand in SF, not solving the problem, but at least reducing it.
Other problems that seem like they exacerbate SF's problems: 1) poor intracity transit, e.g. lack of an east-west subway, concentrates demand in a handful of eastern districts; and 2) large areas of the city are not even mid-rise, but full of two-story houses (partly historical, partly anti-development, partly related to #1).