[basch]

It might follow to say that current LLM;s arent trained to generate papers, BUT they also don't really need to reason.

They just need to mimic the appearance of reason, follow the same pattern of progression. Ingesting enough of what amounts to executed templates will teach it to generate its own results as if output from the same template.

[Eisenstein]

What is the difference between 'reasoning' and 'appearing to be reasoning' if the results are the same with the same input?

[ben_w]

The outputs aren't really the same, they simply seem plausible at first glance.

For example, I recently experimented with using ChatGPT to translate a Wikipedia article, on the grounds that it mighy maintain all the formatting and that Transformer models are also used by Google Translate.

As it was an experiment, I did actually check the results before submitting the translated article.

First roughly 3/4 were fine. Final quarter was completely invented but plausible, including references.

LLMs are very useful tools, I'll gladly use them to help with various tasks and they can (with low reliability but it has happened) even manage a whole project, but right now they should treated with caution and not left unsupervised — Peter principle, being promoted beyond their competence, still applies even though they're not human employees.

[Sakos]

Because the results aren't the same? I use AI every day for software development and a number of other topics. It's very easy to recognize the points where the illusion breaks and how it breaks clearly indicates to me that there's no actual reasoning in the response I've gotten. It often feels like I'm doing the reasoning for the AI and not the other way around.

[nis0s]

From what I’ve seen, the results are not the same. In the latter scenario, there’s a risk of encountering a non sequitur all of a sudden, and the citations may be nonexistent. There’s also no guarantee that what you’re stating is factually correct when your logic is unbounded by reality.