At least in USA, since pretty much the absolute beginning, anyone with any wealth tried to gobble up the absolute most amount of land they could. Both as a form of wealth and a productive input. This has been less so in the past century though as agriculture has become smaller part of the economy.
Put a few mile ring around any city center and yes the land there is scarce. You would not want one or a small number of individuals buying up all or a lot of that land because other people want to live there too. Therefore do exponential taxation based on land they already own. Why is this confusing to you?
There are a large number of properties in big cities sitting empty due to people hoarding land/houses. Too much is anything more than your primary residence. It should be taxed exponentially with both number of secondary residences and size normalized to urban development around the property.