[epidemian]

[author of the post here]

thanks for the very thorough reply! it's fascinating to see the techniques used to improve LLM outputs :)

i'll reply to some specific points, but i think your main argument of trying to find ways of working effectively with LLMs is spot-on.

> try this yourself: generate a novel pangram without any sort of iterative process or revisions -- the first thing that pops into your head you must commit to paper. it's very hard.

yes, it would be indeed very hard. and i don't know why i would try to do it that way. notice also that i didn't instruct ChatGPT to do it that way either.

to generate a pangram, i'd probably start with some random phrase, count the letters i've used and which ones i'm missing, and then iteratively tweak the phrase to use more letters of the alphabet until i've used them all. that at least seems like a reasonable strategy. and i would expect any intelligent agent to do the same. not that particular strategy, but "the same" as in: to try to find a strategy that works. after all, isn't that a fundamental part of intelligence? being able to find solutions to novel problems.

i know that LLMs don't work that way. and that's fine. but that was also the main point i tried to make in the post: we're being sold LLMs as "intelligent", but they don't work in any way like what we would intuitively say it's intelligent.

[hydrolox]

I think that shows how LLMs lack an important part of what we as intelligent agents have, as the parent comment pointed out, the innate ability to have some sort of train of thought or self check mechanism. In a human, as you said, we don't immediately blurt out or write out a phrase, since if someone gave us this problem we would immediately start considering the constraints and possibilities. By contrast, LLMs do not have this and the ability has to be "bolted on" through a pre prompt like "consider if your answer is correct" and "think step by step" etc. As far as actually choosing a viable strategy, such as figuring out to go alphabetically, this probably emerges as they get larger and larger; i.e, they need both the bolted on train of thought ability and also an actual good sense of reasoning and logic, which could be compared to a person (maybe a child) who can't come up with a good solution, let's say, to making a pangram (in this case).