One, a low-latency fallback beats no fallback. Two, at least for Waymo, the system is engineered to be high latency. Back-up drivers seldom directly drive the car, and when they do, it's not at the last minute. Instead, they give high-level instructions the car actuates.