In many countries (an ever-expanding list), you cannot purchase a SIM card without showing ID, and a copy of your ID is made and sent to the state authorities. That is, mobile numbers are always connected with your identity, you cannot have an anonymous phone number. Consequently, the state can easily determine which of its citizens are using Signal.
While communications on Signal are end-to-end encrypted, in authoritarian states merely using a secure messenger can draw police suspicion.
Some people might not want to give a phone linked to them to certain people while attempting to remain anonymous.
I don't disagree this is less then the userbase of Signal, but I assume it shouldn't be hard to use a randomly generated ID (or similar) system as a fallback options for identifiers, and leave phone numbers as the default and recommended setting.
While communications on Signal are end-to-end encrypted, in authoritarian states merely using a secure messenger can draw police suspicion.