While that's true, personally I'm ok with birth place being a restriction for some things e.g. in order to run for President in the US, one needs to be born in the US, not merely a citizen. Not sure how it is in other Western countries.
The problem with this logic is that when applied, it can absolutely be racist.
For example, in your example what happens to those born in say, Puerto Rico? Are they barred from running for president because they weren't born in the US?