Except the local abbot would happily take 1/10 of the fruits of your labor, the landlord even more, and you stood no chance seeking better fortunes away because roads were in disrepair, infested by robbers and patrolled by mercenaries tasked to kill any fugitive...
But you died of famine and disease more often - population went way down. It became harder to trade even the most basic foodstuff, since simple roads to the closest market became unsafe. Eventually feudalism happened and a lot of simple peasants were literally enslaved to the land.