I don't think anyone is saying that. You will just need to be authenticated before giving any commands to the bank. Maybe some type of TOTP that you can use over the phone or in person.
That is the exact problem. You have identification tied to your device. Your device is lost or stolen. Now you can't access your bank account. Human support can help you out by finding flexible ways to ascertain your identity. This is the angle social engineers exploit, tricking employees trying to be helpful to abuse that area of flexibility. You can take away human judgment and all flexibility in the system, and that will make the system more secure, but it also results in a deeply uncaring system that makes life harder for people. Rigid bureacracy doesn't do a good job of accounting for a house fire destroying everything you own or your e-mail provider shutting down; these are fringe cases but they do happen and there are positive resolutions available as long as human discretion is involved.
Credit cards are lost and stolen all the time, and it isn't really a big deal when it happens, since charges can usually be easily reversed. This does not sound like the same scenario. It also doesn't account for people who lack friends/family nearby or at all.
> it breaks and you were too cool to provision a backup
If we're relying on the average person to back things up properly, this idea is doomed from the start.