I hear you, but the not so cool thing about SMS is how trivially easy it is to clone a SIM. My level of trust for information passing through the SMS channel is near-zero.
You're absolutely right - SIM swapping is a major threat in many parts of the world. If you're a target and the attacker is determined, you could be in serious trouble. However, it's more cost-effective for bad guys to launch phishing campaigns using malicious proxies like Evilginx or Modlishka to target the masses.