"Cocked & Locked" is usually how people refer to this - and it is easy to screw up. Under stress, people's fine motor skills vanish, sometimes resulting in the safety not being disengaged as you draw from the holster. Additionally, it can be accidentally flipped off during handling.
Modern firearms have multiple internal safeties to prevent accidental discharges (unless you're Sig apparently).
If for some reason you're open carrying in a holster (like perhaps a police officer or soldier would be), and someone tries to get your weapon off of you and succeeds, a manual safety could save your life. They probably won't realize that the safety is on, and when they point the weapon at you and pull the trigger, nothing will happen, giving you a chance to escape or fight back.
Without that manual safety, the weapon just goes off and you now have an aftermarket hole installed in your body.
There's less of an argument manual safties in concealed carry, though. The opponent shouldn't know you have the weapon until it's drawn, so there's less chance of them getting it out of a holster.
Unfortunately the 2A community is filled with fud and hollywood-esque beliefs. Those who have little training or practical experience often fall victim to believing these fantastical "what-if" scenarios.
The amount of times a firearm has been taken out of someone's duty holster and used against them is a statistical rounding error. Most LEO departments require Level 3 Retention holsters, making this scenario nearly an impossibility. Additionally, a safety isn't going to save someone's life if the "bad guy" has wrestled control of the firearm away from the owner.
This entire scenario is fiction. Manual safeties often do more harm than good - which is why just about all modern handguns have no manual safety (outside of the few States which, out of ignorance, require them by law).