[esjeon]

> They are blackboxes for the normal user the same way as a smartphone is a blackbox.

You can't take that approach. The current NN techniques are blackbox-by-nature, and are blockbox to everyone including the devs. Proprietary software is only blackbox to consumers, and large complex software still have insides that can be observed when things go wrong. For NNs, nothing can describe how exactly they work, and each network has to be reverse engineered independently, which is, AFAIK, a separate research field.

> I can't drive 10 Million KM in my lifetime (i think). The cars from Google and Tesla already did.

The length (nor the amount of data) alone doesn't decide the quality of AI. Actually, ROI diminishes rather quickly in the early stage of development. The rest is about picking up corner cases. They can drive 1 parsec, and still would not perform better than the ones we have now.

Also, again, because NN is a complete blackbox, even the devs can't be sure if those corner cases are properly reflected to a newly trained network, nor if the new training data didn't impact the performance in other corners. We just don't know for sure. We just take chances. That's the limitation.