[Androider]

The equivalent for a human would be an reflexive response to a question, the kind you could immediately answer after being woken up at 3am in the morning. That type of answer has been deeply trained into the human networks and also requires no deep insight.

But if a human is allowed time and internal reasoning iterations, so should the LLM when determining if it has deep insight. Right now we're simply observing input -> output of LLMs, the equivalent of snap answers from a human. But nothing says it couldn't instead be an input -> extensive internal dialogue, maybe even between multiple expert models for seconds, minutes or hours, that are not at all visible to the prompter -> final insightful answer. Maybe future LLMs will say, "let me get back to you on that".

[lukev]

Completely agree.

From a computer science point of view: a single prompt/response cycle from a LLM is equivalent to a pure function; the answer is a function of the prompt and the model weights and is fundamentally reducible to solving a big math equation (in which each model parameter is a term.)

It seems almost self evident that "reasoning" worthy of the name would involve some sort of iterative/recursive search process, invoking the model and storing/reflecting/improving on answers methodically.

There's been a lot of movement in this direction with tree-of-thought/chain-of-thought/graph-of-thought prompting, and I would bet that if/when we get AGI, it's a result of getting the right recursive prompting pattern + retrieval patterns + ensemble models figured out, not just making ever-more-powerful transformer models (thought that would certainly play a role too.)

The LLM isn't the whole brain. Just the area responsible for language and cultural memory.

[kaba0]

Ideally a recursive execution would also be a pure function - maybe a better way to put it about current LLMs is that they are a single mathematical expression being built up from a fix number of nodes and only addition and multiplication.

[lukev]

yes, except the "reasoning" process should also be able to look up facts (retrieval) and invoke external tools, making it non-pure.

[Scubabear68]

Agree. I am not an expert on AGI, but I suspect this route is a dead end because I believe true AGI requires interaction with an external environment to learn eg there needs to be a feedback loop to external things that it can interact with and learn from.

“You learn best from your mistakes” comes to mind here.

And not just shallow feedback loops but bigger, more profound ones eg “I didn’t go to work for a month and so I was fired and and now I have no money and it will now be hard to be hired again”.

[hskalin]

Do you have any good resources for chain-of-thought prompting experiments?

[I_Am_Nous]

This reminds me of The Last Question by Isaac Asimov. I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways. Similar to the safeties already in place in LLMs, but instead of it just saying "NO DON'T SAY THAT" it can dialog internally to change what the output is until it reaches what it believes to be the agreed upon best answer.

[mr_toad]

> I also think if we stopped expecting all LLMs to have an immediate answer, it would be relatively easy to shim some kind of "conscience" to direct the output in different ways.

If the shim was just another AI, then how do you align that AI? Who watches the watchers? But if it was a deterministic algorithm it would probably fail for the same reasons that algorithmic AI never went anywhere.

[I_Am_Nous]

A great point! A smaller AI with a rather limited parameter count could be trained for individual needs so some things (chat moderation) might be easier to do than other things (fact check peer reviewed papers in a verifiable way). For some use cases it would be overkill to have a conscience but an AI spokesperson for a company will probably have a company-aligned conscience for obvious reasons.

[lifeisstillgood]

It would have an emotional reaction to certain "thought constructs" and would be guided by that.

Or we could just give them three laws

[I_Am_Nous]

With current LLMs the three laws might be tough to implement in a way that can't be prompt injected around. That's why I described the extra bits as a "conscience" which could enforce the three laws. Maybe the three laws are the internal conscience's context prompt while the main LLM is more able to think anything in general and then the output is tuned down by the conscience?

Otherwise the laws will have to be implemented as weights or during training so the model explicitly knows the laws and would never even be capable of doing something against them.

[pixl97]

I mean, the whole purpose of the I, Robot story was to show you that the 3 laws didn't work. We had the first story on prompt injection decades ago and we just didn't realize it.

[ShamelessC]

Indeed. Asimov says as much in interviews - the laws are flawed from the get go. It is the naive belief that the laws work by various scientists that creates these Sherlock-style mysteries/crimes to be solved in the first place.

I suppose the real wisdom there is that humans are doomed to fail at alignment if we create sentience and expect it only to serve us.

[TeMPOraL]

Obligatory reminder that the "three laws" were invented to be deconstructed, with Asimov spending a lot of pages showing many ways in which they completely fail, illustrating that AI alignment is a hard problem.

[jvanderbot]

I think on of the reasons we require so much data is that we try to bake all that "simulated experience and internal dialogue" into the snap responses. I bet if you could do an efficient sim/test retraining, you'd do data-driven responses on the fly.