In a capitalist dystopia, I'd agree with you. But everywhere else, you cannot live without impacting the lives of other people.
For example, let's say you choose to eat unhealthy food, become obese, and land in the hospital. You might say "who cares what other people eat" while you are using a limited, shared resource.
If you're able to not work indefinitely because you're consuming a limited, shared resource like the money of other people that live in a country, why should they think that's fair if it's supposed to be a safety net?
Depends. Most are not, but in some cases, they are (e.g. islamic terrorism in France and Germany).
But the main reason is that some peoples/places have developed their own culture over centuries, and don't particularly appreciate some large rush of newcomers bringing their own stuff and messing the nice thing they have got going.
In Europe, you're more likely to die of lung cancer than terrorist attack. It's a load of crap that you'll meet anyone in your life who will genuinely try to end your life. Sure, you might be unlikely, but you also might win the lottery.
With respect to culture.... I don't know what to say to that. It strikes me as very xenophobic and more than a little naive. The way to get your culture to remain is by showing others what is to celebrate about it, not by forbidding them from living around you. It's not like any culture can exist long unchanged while children have access to the internet anyway.