It’s not as simple as “the driver can take over if they need to.” In many cases it takes a human too long to recognize the system doing something dangerous and to intervene. This is going to end badly.
Indeed, recognizing the problem takes vital time and taking over to execute a maneuver is way harder than executing the same maneuver while you're already in control.