When you ask an AI for a list of citations and it returns a list of 10 names, pages and dates no amount of 'critical thinking' will let you determine which answers are correct or incorrect, you have to sit there and painfully cross-reference each one.
The 'critical thinking' part at this point in time leads you to only one conclusion: that the tool is unreliable and dangerous and shouldn't be used. Or, you're teaching kids that they need to cross-reference all 'authoritative sources' which leads to a complete failure of the system - if you give kids a textbook will they have been 'trained' by GPT to not believe a word of any of the claims in it without cross-referencing? How does that enable better education.
If you provide a tool from a position of authority it needs to be a reliable and believable tool; standing in a position of authority and handing out access (to children!) to a lie box and asking those children to 'rely on it' is frankly abusive. That's beyond mixed messaging, that's just some form of weird mental torture.
But if the second and third and fourth source exists (let us suppose the 5th doesn't) what will the kids trust, the AI or the search engine? We all know that google is becoming crap and high schools on average don't have a 'proper' library so... Will they assume the first source exists but isn't indexed?
I just think it's a real issue to give kids free access to a tool that so confidently makes stuff up. I know that textbooks aren't perfect but they're at least audited by a committee of experts, mistakes slip in but there's oversight, there's a procedure to ensure that they're reliable representations of (currently understood) facts, and that when politics enters the textbook production process it (correctly) becomes a bit of a national scandal.
I hope for that too. But I believe it wont. The complete Fake News topic just shows that everyone is happy trusting information that matches their beliefs. There is never any critical thinking or verification if it is actually true or just made up.
And its enough when such a technology is 80% right. Nobody will verify every answer if it is correct most of the time.
Although kids necessarily don't have those ego dependent beliefs ingrained within them yet. So if they see that false information can be presented very convincingly, perhaps they learn not to trust just everything and in addition question what their parents are telling them.
The issue arises when the belief you have is tied to your ego so you only cherry-pick the evidence that will satisfy your ego and ignore/question everything else, but not this.
Adults accept all manner of weird claims generated by humans on Facebook, television or in their mailbox as fact, especially if said confidently and authoritiatively. So while I expect many kids to also accept hallucinations as fact, I am not sure that is a much different world.
Same as with every piece of information in the past. How do people react to information now? If the source seems to be trusted, or confirm existing biases and expectations, they will likely believe it.