If you care about free markets or a smaller state, but not the free movement of labor at the same time, you are holding hypocritical beliefs.
Every nation on Earth has border restrictions of some kind. Though recent in human history, such restrictions clearly have utility and value in the modern world.