[chaosist]

I have thought a lot about this and I suspect "I don't know" would be devastating to the model.

The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :)

I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.