People drive to work anyway. A vacation or two every year is probably not even a double digit percent of a person’s total fossil fuel usage, and gives them a lot of happiness and reason to work and do things that are good for society.
Why is there a segment of the population that wants to live in poverty and squalor?
How much pollution is okay? Why not argue for efficiency standards rather than bans?
Everything could be said to “harm people”. Banning travel could make some people depressed and who knows what that could lead to? Or it might lead to a less connected world and less familiarity with people in other places, and maybe makes wars more likely?