Yeah it would definitely still be a problem, but history shows that life finds a way. Even if everyone has to eat nothing but planted potatoes from any patch of grass that one can lay eyes on.
What history has taught us is that life finds a way by staying together and each person having their function within society, only some of which is growing or producing food.