[jlmb]

I think the main problem is that SMS sender numbers can be easily spoofed (might depend on country, operator, …), so relying on “this message came from where it says it came from” is not really possible.

It might not be an issue for some types of usage, but sounds risky if used for account security/recovery/etc.

[evan_]

If you keep your number private it won’t matter. In fact you could spoof the number on purpose for an extra layer of security.

[jlmb]

Phone number verification (of any kind) is supposed to make sure that the phone number provided belongs to the account owner.

If the number is not actually validated in a secure (enough) manner, there's no point in using phone numbers at all.