The idea I had of London as a place I could live for decades when I first moved here: that's starting to fade away. That's the fall. And it's not just me; it's lots of other EU citizens.
Stop being so literal. Rome is still there, and has been for 2000+ years. It just isn't a terribly important city any more, globally; London will be similar.
London is the idea that a city can have and respect both a past with all its cultural heritage merits/baggage/identity as well as a future as a globalized hotspot. The corporeal hope that you don't have to give up one to have the other. If post-Brexit London turns out to fail on the globalization side of that balancing act, that hope takes a hit not just for Londoners but for everyone who values both sides.
It's the hope that is feared to fall, not the bricks and stones.