Yeah, I fear a lot of human exuberance (and thus investment) is riding on the questionable idea that a really good text-fragment-correlation specialist engine can usefully impersonate a generalist "thinking" AI without doing too much damage. ("LLM, which rocks are the best to eat?")
But there's a scarier further step: When people assume an exceptional text-specialist model can also meta-impersonate a generalist model impersonating a specific and different kind of specialist! ("LLM, create a legal defense.")
They’re plausible word sequence generators, not ‘planning for the future’ agents. Or market analyzers. Or character evaluators. Or anything else.
And they tend to be really ‘gullible’.
What evidence do you have they could do any of those things? (And not just generate plausible text at a prompt, but actually do those things)