Some recent studies have shown that a properly implemented LVT avoids harming most homeowners, in fact you may come out ahead if other taxes are eliminated. (LVT alone is adequate to fund government spending and entitlements.) The homeowners who will typically feel a pinch are those who have inherited very large estates which don’t generate income. They may need to sell or develop part of their estate.
Land speculators, however, will be in serious trouble.
https://landeconomics.org/reports/spokane-report
I think what you're missing is that land in the wealthiest parts of town is worth exponentially more than land out in the suburbs