[NoGravitas]

The thing holding AI back is that LLMS are not world models, and do not have world models. Being confidently wrong is just a side effect of that. You need a model of the world to be uncertain about. Without one, you have no way to estimate whether your next predicted sentence is true, false, or uncertain; one predicted sentence is as good as another as long as it resembles the training data.

[mojuba]

In other words, just like with autonomous driving, you need real world experience aka general intelligence to be truly useful. Having a model of the world and knowing your place in it is one of the critical parts of intelligence that both autonomous vehicle systems and LLM's are missing.