[entee]

I think olivewell's comment in this subthread is what dooms the idea of camera-only self driving for a really long time horizon: you basically need general AI for it.

I work in machine learning in a different area (healthcare) so my perspective may be incomplete, but what I see in ML/AI is that models are really good at memorizing, not so good at understanding. What I mean by that is that a human navigates the world by knowing what a car is, what fog is, what a person is, what a plastic bag is, all sort of things. Object detection can get much of the way there, whether it can get enough of the way there is an open question, but ok we'll grant that. Whether AI/ML can actually make the step from knowing what the object is to knowing how that object interacts with all the other objects in the world is another.

In other words identifying an object is just step one, understanding what that object means in context is the next absolutely required step, and it's way harder. We have a leg up as humans so we can rely on visual alone, but machines almost certainly will need other sensors that report additional information because they don't "get" context in the same way. And I'd say it's an open question whether even with those sensors, they'll be able to get there. I hope so, but the game is far from won.