[RugnirViking]

No. It will give you a long answer with correct grammar, punctuation, use of wide vocabulary, and persuasive sounding arguments. What we are learning nowadays is that among humans, that is highly correlated with highly intelligent, substantive answers from intelligent, practiced subject matter experts.

Among AIs, such text is merely correlated with having read a lot of literature. It's sometimes right. It's sometimes wrong. But you don't know, and any attempt to defer to "oh well it sounds persuasive", which may have served you okay with smart humans, will end up failing in spectacular and unpredictable ways.

I do not say this because I don't find AI's interesting, or even useful. They are, for tasks they are suited too. But there are so many people essentially arguing they are suited to all tasks, which they clearly aren't

[DiogenesKynikos]

You are seriously underselling what LLMs do nowadays. It's not just that the grammar is correct. In most cases, the answers are substantive and factually correct.

You can ask fairly complicated questions, and it will usually reason correctly and give you a high-quality answer. I ask about programming, physics and math, and it usually answers on the level of someone with a high level of training in those fields.

It sometimes fails in strange ways, but you can't just write off all of the high-quality answers LLMs give as nothing more than plausible-sounding English.