You can have a say over someone’s property by purchasing it. Taking away valuable uses of someone’s property (where the use doesn’t harm anyone else’s property) without compensation is stealing.
I think Penn Central was a fantastically wrong decision (Rehnquist and Stevens dissented in that case). Obviously, the fact that a regulation diminishes the value of property is not sufficient to amount to a taking. The government can ban diesel cars and if that renders existing diesel cars worthless, so be it. Height limits (and the concept of air rights trading) can be justified on a similar principle, but they’re closer to the borderline. But historical landmark laws are not laws of general effect that happen to impact property values. They are targeted takings of private property without compensation.
The landmark laws have had essentially reverse blockbusting/gentrifying effects in poor neighborhoods in New Orleans like Tremé. It’s not uncommon for speculators to call in violations like broken stoops or split porch railings and then the same speculator leaves flyers in the mailbox of the “violator” offering to buy for a knockdown price...
https://www.theguardian.com/cities/2015/jul/02/calcutta-arch...