[mikorym]

You don't need it, but sets are more primitive than numbers.

For example, the category of finite sets without an NNO [1] is simpler or more foundational than the set of natural numbers. At the same time, this category is actually a category of numbers.

My point is that numbers are complicated by nature, but they are more intuitive for humans (mostly I think) than sets are. Sets are simpler or more basic, but often less intuitive for humans.

[1] https://en.wikipedia.org/wiki/Natural_numbers_object