I think this is a generation gap thing. Most people under 30 don’t call or text anyway, so a phone number feels like pretty useless and low priority personal information.
I don't call or text much, but I don't appreciate how phone numbers cost money and are intrinsically tied to your personal identity (unless you're an adult and spend time and money shopping for burner phones), whereas email addresses and Matrix accounts are permissionless.
The phone number being tied to your identity is the entire point. If you get banned from a community are you really going to go to Verizon and ask for a new one every time you want to join again? The alternative is the kind of problem that is all too common on IRC where a troll keeps making burner accounts faster than the mods can ban them and generally ruining the experience for everyone else in the channel.
I imagine this is geographic. Texts are quite expensive in some places, and that means people use alternatives instead. Also some people had pervasive mobile phone usage for a long time before smartphones made other options available, which I imagine would add to their popularity. Here in Denmark SMS is very entrenched, but I know it varies in other countries.