UAVs don't have to deal with traffic (the thought of driving a vehicle with the latency and intermittent connectivity of my drone horrifies me) and when someone dies in a video game they respawn...
It might not actually matter. Since the car can operate autonomously already, the operator doesn't necessarily need to literally drive the car. They might simply need to hop in to verification of actions in unusual situations.
I'm imagining a situation where a car comes across a parked truck on a one-way road (common in cities). A human operator comes in the loop to ensure that it's actually safe to switch lanes and pass. Check for things like emergency vehicles, unusual pedestrians, etc. They don't need to literally take the wheel, just confirm that the vehicle can take a specific action.
Getting a remote driver connected might a while, but afterwards it seems like a mostly solved (in practice) problem.