[operatingthetan]

>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem.

The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.

[aetherson]

You're confusing the training method with the internal process. If I had you repeatedly attempt to learn how to make believable completions of partial documents about a given topic, you would eventually learn things about that topic and could use your knowledge to create more believable completions of documents about that topic.

[goodpoint]

believable != true

[operatingthetan]

LLMs do not learn. You put it out to pasture and create a new one. "Memory" in a session is essentially a context window party trick.

[lemiffe]

The LLM itself doesn't, but agents can research, compare, add to their memory, and use that to narrow the results down to a probabilistically higher set of outputs; I have used an LLM for my own MRI results and it was nearly spot-on, verified by a subsequent visit to a specialist. YMMV as they say. But I do believe we are entering the era where LLMs are considering past interactions and long context windows to inform it of personal preferences and history in order to output more accurate results.

[aspenmartin]

They do learn in context, and very sample efficiently. Continual learning is active area of research and we sort of already have something resembling it with persistent context. So yes they do learn.

[operatingthetan]

I consider that to be the illusion of learning. You are not wrong, I think they may actually learn in the future though. But not today.

[singpolyma3]

That presumes that we have a definition of "thinking" or that we know that anything is "thinking" when in fact neither is true.

The problem is real but I don't think positing a philosophical root is helpful

[operatingthetan]

The claim that we are assigning human-like agency to a machine with none is simple and factual.

[ForceBru]

What's "thinking"? What's "agency"? What's "human-like agency"?

If "agency" is making decisions and performing corresponding actions in the real world, then LLMs most definitely LOOK LIKE they're making decisions (what's the next token? which tool to use? what's to say, in general? what idea to convey?) and performing actions (tool use). Can we tell whether they are ACTUALLY making decisions? Well, are the people around me "actually" making decisions? Or are they simply pushed around by circumstances and external forces?

Am I actually making decisions? Did I like DECIDE to write this comment? Maybe? I have no clue...

[operatingthetan]

I think you're mildly obfuscating the issues at hand by diving too deeply into philosophical questions.

It's quite simple, the agency that the LLM appears to have is actually your own. Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems.

[aspenmartin]

You are implying definitions that don't seem to be mainstream; thinking is internally manipulating information to reason, infer, plan, solve problems, and form judgments or beliefs. Also -- "Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems." it sounds like you paint this like it's something fundamental? It isn't. Nothing is stopping you from streaming information to an LLM and letting it process this information, this is precisely what people are trying to build.

[singpolyma3]

The idea that humans have agency is supernatural thinking imo

[operatingthetan]

A free will versus determinism argument doesn't really have a place here. Consider instead that humans factually have 'the illusion of agency.' The LLM does not even that have that. It cannot act on it's own, it has no ongoing drama or intention. It only reacts to prompts.

[corndoge]

Often times the words produced do have legitimate factual information though. It's less psychosis and more a confluence of well known human tendencies - salience bias, automation bias, etc.

[operatingthetan]

The big problem is often times they don't as well. That's why we can't rely on them.

[aspenmartin]

Same with humans? Doctors, scientists...if a tool has any error rate above zero its not reliable?