You don't need it, but sets are more primitive than numbers.
For example, the category of finite sets without an NNO [1] is simpler or more foundational than the set of natural numbers. At the same time, this category is actually a category of numbers.
My point is that numbers are complicated by nature, but they are more intuitive for humans (mostly I think) than sets are. Sets are simpler or more basic, but often less intuitive for humans.