At least in USA, since pretty much the absolute beginning, anyone with any wealth tried to gobble up the absolute most amount of land they could. Both as a form of wealth and a productive input. This has been less so in the past century though as agriculture has become smaller part of the economy.
Put a few mile ring around any city center and yes the land there is scarce. You would not want one or a small number of individuals buying up all or a lot of that land because other people want to live there too. Therefore do exponential taxation based on land they already own. Why is this confusing to you?
A house used to be viewed as a consumable good a hundred years ago, not an appreciating asset. (Before the New Deal commoditized mortgages)