Freedom of movement is great, unfortunately we live in a world with 7 billion people and it's likely a vast majority would move from where they are at if they could. 1.3 billion people make less than $1.25 a day. What do you think would happen if they could move somewhere else easily? Equality and economic parity is necessary without creating a humanitarian and environmental disaster of human movement.
I do too, that's where elite interest and interest of modern workers converge and they use that to justify their policies. But often those policies are not good for the locals. Many immigrants do work harder than the locals. Which creates more competition and allows you to exploit the local population better in terms of offering them lower pay and longer hours and fewer benefits. Combine that with automation and you really have too much power in the situation.