[heavyset_go]

It's given me blatantly wrong results for things that come up easily on Google, and it tells me those wrong things with confidence.

What's worse is that someone who isn't a domain expert might end up being convinced by the arguments and conviction the model provides.

Your second question is a good one, and I see that as a big problem with this generation of AI/ML. You're starting to scratch the surface of "this requires understanding of the real world" problems without models really understanding anything. It's all statistics and correlations in data, the model is not capable of really understanding its input or output.

These models are like very impressive Markov chain text generators in that they can spit out pretty convincing answers that seem cogent, but there is no real comprehension going on with what they read and write. It's just statistics.