[erispoe]

San Francisco is largely made up of renters (63%) and planning decisions are made by the City of San Francisco, not individual neighborhoods. The problem you describe exists in small communities where the neighborhood is de facto a jurisdiction. Like Brisbane, right south of SF. In San Francisco, NIMBY homeowner are actually a minority playing against the interests of the majority of the citizens of this city.

Now there are several causes for this. A major one is concentrated harm vs distributed benefits for specific development. A new development in a residential neighborhood is gonna piss off a small number of people and motivate them to organize to put pressure on the planning commission. This development would benefit pretty much everyone else, but in a way that is so distributed that it's really hard to motivate people to fight for it. A new development is gonna have a very concentrated impact on neighbors and a very diluted effect on the rest of the city.