That’s kind of like saying that because I have a house on a tiny plot of land and my rich neighbor owns half the land in town, that property taxes don’t affect me. It doesn’t really follow.
It certainly woudn't affect you as much as your rich neighbour. And if the additional tax on land brought in more community services, you would be all for it, whilst your neighbour would be against (since he'd be paying a majority of that cost to fund said community service).