It is very difficult to build anything in SF. The city doesn't have any available land, therefore you need to demolish existing structures and build higher. There is a loud and vocal community of activists that refuse to allow new construction because of gentrification. Additionally, many won't allow retail businesses that are not local. SF prevents retail chains of a certain size in some neighborhoods. Also, the city has a law that allows any resident to challenge a change of use permit or new construction, no matter where they live in the city. These cases ultimately come up before the Board of Supervisors which tend to attach requirements and restrictions for construction leading to expensive delays.
Japan has zoning laws at the country level, and they are much more sane than local zoning laws. Basically, local zoning says "this piece of land can be used for one thing". Japan's laws say "this piece of land can cause a maximum nuisance of X, but it can have all these different things on it". And then it's up to the developers to decide if building a multi-unit building would more profitable than a single family home.