Verification is easy, just like with humans. It's called a driver's license. And since the AI does not tire and can probably be sped up, you can put it through hundreds of thousands of hours of driving quickly in a good simulator with adversarial and normal situations, then rate it at various tasks. Just like we should do with human drivers, but fail to.
Explanation is harder. But we probably shouldn't care, even in courts people cannot often explain what and why they did while driving, or they just lie. The thing is, for liability purposes you have to ensure it is not a series defect and that a good human driver would not be able to handle it
But we evolved to have empathy to help understand how humans act in the cases we don't have complete or good information.
This is why "crazy" people make us so uneasy. They don't fit our mental models for how a human should act. Would you be comfortable driving with a road full of unpredictable "crazies"?
I wonder if we'll ever have the same level of trust with AI as humans if it is still being used at a black box level.
Bad AI will suffer from the Dunning-Kruger effect and overestimate its abilities, while good AI will suffer from Imposter Syndrome and underestimate its abilities.
Explanation is harder. But we probably shouldn't care, even in courts people cannot often explain what and why they did while driving, or they just lie. The thing is, for liability purposes you have to ensure it is not a series defect and that a good human driver would not be able to handle it