Implementation flaws like that are always possible, but my concern is that in so many cases, SIM swaps are ridiculously easy by design (or more accurately, by absence) of the phone provider's security procedures.
Issue is that FCC mandates a port out within 4 hours and stores don't make $$ while doing these so their goal is to get you out of the door ASAP so they can focus on the revenue. So that's why + bribe factor