You get taxes in other ways through new building permits, higher property taxes, local sales tax, etc...there is more going on than just collecting corporate taxes in a municipality. The more people you draw the better your overall tax revenue is...and the taxes you are willing to put off are limited in time and generally expire after 5-10 years.
It's not really a problem for the partners in the joint private/public venture. It is a problem for the next jurisdiction that is now facing companies' expectations of massive incentives, for the existing residents and businesses that are paying normal taxes because they aren't new, for any but the largest businesses who don't have lobbyists on staff, etc.
There's also the general wisdom that life is better when you're competing to offer more value for more money, not just cutting costs as far as you can
I see your point and agree that from the fairness point of view this is not quite right. However, I do not see fairness as relevant in this context. Unless it is Walmart that is getting tax breaks for building a bunch of stores in NYC, local businesses do not oppose big companies getting incentives to move in.
Perhaps they don't but I'm not as focused on the relationship between the company and jurisdiction, I'm talking about the externalities.
Desperate places give away the store to get anything at all, and it depresses what companies are willing to offer the next place while also tilting the playing field toward big companies with the ability to negotiate these kind of deals.