Very true. There are many areas in the US (Detroit for example) that have lost population over the past 50 years that wouldn't have to start from scratch with new development.
The homes from detroits population high did not sit vacant and maintained over the decades. They rotted apart and many were razed. Much of inner city detroit really would be like starting from scratch considering the amount of vacant greenfield land available.
Assuming infrastructure serving vacant lots were maintained by a famously insolvent city budget is a big one to make. Sometimes replacement or retrofit are vastly more expensive than greenfield development.