Underutilization of land as a result on speculation on the growth of land prices is a major reason, even Manhattan has massive chunks of real estate that are vacant, a land value tax would fix this.
That's an effect of massive demand, not a cause (or not a first-order cause). If you know (and vote for) loads more demand is coming the more you wait, you will wait. If demand were more limited to natural population growth levels, there'd be no point waiting.