[paxys]

Don't be sycophantic. Disagree and push back when appropriate.

Come up with original thought and original ideas.

Have long term goals that aren't programmed by an external source.

Do something unprompted.

The last one IMO is more complex than the rest, because LLMs are fundamentally autocomplete machines. But what happens if you don't give them any prompt? Can they spontaneously come up with something, anything, without any external input?

[BeetleB]

> Disagree and push back

The other day an LLM gave me a script that had undeclared identifiers (it hallucinated a constant from an import).

When I informed it, it said "You must have copy/pasted incorrectly."

When I pushed back, it said "Now you trust me: The script is perfectly correct. You should look into whether there is a problem with the installation/config on your computer."

[TSUTiger]

Was it Grok 4 Fast by chance?

I was dealing with something similar with it yesterday. No code involved. It was giving me factually incorrect information about a multiple schools and school districts. I told it it was wrong multiple times and it hallucinated school names even. Had the school district in the wrong county entirely. It kept telling me I was wrong and that although it sounded like the answer it gave me, it in fact was correct. Frustrated, I switched to Expert, had it re-verify all the facts, and then it spit out factually correct information.

[paxys]

That's the flip side of the same symptom. One model is instructed to agree with the user no matter what, and the other is instructed to stick to its guns no matter what. Neither of them is actually thinking.

[ACCount37]

Wrong. The same exact model can do both, depending on the circumstances.

[logifail]

There was a time when we'd have said you were talking to a sociopath.

[IanCal]

> Don't be sycophantic. Disagree and push back when appropriate.

They can do this though.

> Can they spontaneously come up with something, anything, without any external input?

I don’t see any why not, but then humans don’t have zero input so I’m not sure why that’s useful.

[zahlman]

> but then humans don’t have zero input

Humans don't require input to, say, decide to go for a walk.

What's missing in the LLM is volition.

[dragonwriter]

> Humans don't require input to, say, decide to go for a walk.

Impossible to falsify since humans are continuously receiving inputs from both external and internal sensors.

> What's missing in the LLM is volition.

What's missing is embodiment, or, at least, a continuous loop feeding a wide variety of inputs about the state of world. Given that, and info about of set of tools by which it can act in the world, I have no doubt that current LLMs would exhibit some kind (possibly not desirable or coherent, from a human POV, at least without a whole lot of prompt engineering) of volitional-seeming action.

[jmcodes]

Our entire extistence and experience is nothing _but_ input.

Temperature changes, visual stimulus, auditory stimulus, body cues, random thoughts firing, etc.. Those are all going on all the time.

[goatlover]

Random thoughts firing wouldn't be input, they're an internal process to the organism.

[jmcodes]

It's a process that I don't have conscious control over.

I don't choose to think random thoughts they appear.

Which is different than thoughts I consciously choose to think and engage with.

From my subjective perspective it is an input into my field of awareness.

[zeroonetwothree]

Your subjective experience is only the tip of the iceberg of your entire brain activity. The conscious part is merely a tool your brain uses to help it achieve its goals, there's no inherent reason to favor it.

[IanCal]

LLMs can absolutely generate output without input but we don’t have zero input. We don’t exist in a floating void with no light or sound or touch or heat or feelings from our own body.

But again this doesn’t see to be the same thing as thinking. If I could only reply to you when you send me a message but could reason through any problem we discuss just like “able to want a walk” me could, would that mean I no longer could think? I think these are different issues.

On that though, these see trivially solvable with loops and a bit of memory to write to and read from - would that really make the difference for you? A box setup to run continuously like this would be thinking?

[ithkuil]

It's as if a LLM is only one part of a brain, not the whole thing.

So of course it doesn't do everything a human does, but it still can do some aspects of mental processes.

Whether "thinking" means "everything a human brain does" or whether "thinking" means a specific cognitive process that we humans do, is a matter of definition.

I'd argue that defining "thinking" independently of "volition" is a useful definition because it allows us to break down things in parts and understand them

[BeetleB]

> Humans don't require input to, say, decide to go for a walk.

Very much a subject of contention.

How do you even know you're awake, without any input?

[esafak]

I would not say it is missing but thankfully absent.

[jackcviers3]

The last one is fairly simple to solve. Set up a microphone in any busy location where conversations are occurring. In an agentic loop, send random snippets of audio recordings for transcriptions to be converted to text. Randomly send that to an llm, appending to a conversational context. Then, also hook up a chat interface to discuss topics with the output from the llm. The random background noise and the context output in response serves as a confounding internal dialog to the conversation it is having with the user via the chat interface. It will affect the outputs in response to the user.

If it interrupts the user chain of thought with random questions about what it is hearing in the background, etc. If given tools for web search or generating an image, it might do unprompted things. Of course, this is a trick, but you could argue that any sensory input living sentient beings are also the same sort of trick, I think.

I think the conversation will derail pretty quickly, but it would be interesting to see how uncontrolled input had an impact on the chat.

[awestroke]

Are you claiming humans do anything unprompted? Our biology prompts us to act

[paxys]

Yet we can ignore our biology, or act in ways that are the opposite of what our biology tells us. Can someone map all internal and external stimuli that a person encounters into a set of deterministic actions? Simply put, we have not the faintest idea how our brains actually work, and so saying saying "LLMs think the same way as humans" is laughable.

[triclops200]

As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this applies to llms as well, they easily satisfy the requirements to be considered the mathematical object described as a markov blanket).

And, yes, if you could somehow freeze a human's current physical configuration at some time, you would absolutely, in principle, given what we know about the universe, be able to concretely map input to into actions. You cannot separate a human's representative configuration from their environment in this way, so, behavior appears much more non-deterministic.

Another paper by Friston et al (Path Integrals, particular kinds, and strange things) describes systems much like modern modeling and absolutely falls under the same action minimization requirements for the math to work given the kinds of data acquisition, loss functions, and training/post-training we're doing as a research society with these models.

I also recommend https://arxiv.org/abs/2112.04035, but, in short, transformer models have functions and emergent structures provably similar both empirically and mathematically to how we abstract and consider things. Along with https://arxiv.org/pdf/1912.10077, these 4 sources, alone, together strongly rebuke any idea that they are somehow not capable of learning to act like and think like us, though there's many more.

[stavros]

Thanks for injecting some actual knowledge in one of these threads. It's really tiring to hear these non-sequitur "oh they can't think because <detail>" arguments every single thread, instead of saying "we just don't know enough" (where "we" is probably not "humans", but "the people in the thread").

[triclops200]

Of course, just doing my part in the collective free energy minimization ;)

[goatlover]

> And, yes, if you could somehow freeze a human's current physical configuration at some time, you would absolutely, in principle, given what we know about the universe, be able to concretely map input to into actions. You cannot separate a human's representative configuration from their environment in this way, so, behavior appears much more non-deterministic.

What's the point in making an argument in principle for something that's not feasible? That's like arguing we could in principle isolate a room with a physicist looking inside a box to see whether the cat is alive or dead, putting the entire experiment is superposition to test Many Worlds or whatever interpretation.

[triclops200]

Because that's how the rules of the system we exist within operate more generally.

We've done similar experiments with more controlled/simple systems and physical processes that satisfy the same symmetries needed to make that statement with rather high confidence about other similar but much more composite systems (in this case, humans).

It's more like saying, in principle, if a bridge existed between Mexico and Europe, cars could drive across. I'm not making any new statements about cars. We know that's true, it would just be an immense amount of effort and resources to actually construct the bridge. In a similar vein, one could, in principle, build a device that somehow stores enough information at some precision needed to arbitrarily predict a human system deterministically and do playback or whatever. Just, some levels of precision are harder to achieve than others in terms of building measurement device complexity and energies needed to probe. At worst, you could sample down to the uncertainty limits and, in theory, reconstruct a similar set of behaviors by sampling over the immense state space and minimizing the action potential within the simulated environment (and that could be done efficiently on a large enough quantum computer, again, in principle).

However, it doesn't seem to empirically be required to actually model the high levels of human behavior. Plus, mathematically, we can just condition the theories on their axiomatic statements (I.e., for markov blankets, they are valid approximations of reality given that the system described has an external and internal state, a coherence metric, etc etc), and say "hey, even if humans and LLMs aren't identical, under these conditions they do share, they will have these XYZ sets of identical limit behaviors and etc given similar conditions and environments."

[logifail]

> Yet we can ignore our biology, or act in ways that are the opposite of what our biology tells us.

I have Coeliac disease, in that specific case I'd really love to be able to ignore what "my biology" tells my body to do. I'd go eat all the things I know wouldn't be good for me to eat.

Yet I fear "my biology" has the upper hand :/

[iammjm]

Good luck ignoring your biology’s impulse to breathe

[hshdhdhj4444]

You think an LLM cannot switch itself off?

[gwd]

> The last one IMO is more complex than the rest, because LLMs are fundamentally autocomplete machines. But what happens if you don't give them any prompt? Can they spontaneously come up with something, anything, without any external input?

Human children typically spend 18 years of their lives being RLHF'd before let them loose. How many people do something truly out of the bounds of the "prompting" they've received during that time?

[khafra]

Note that model sycophancy is caused by RLHF. In other words: Imagine taking a human in his formative years, and spending several subjective years rewarding him for sycophantic behavior and punishing him for candid, well-calibrated responses.

Now, convince him not to be sycophantic. You have up to a few thousand words of verbal reassurance to do this with, and you cannot reward or punish him directly. Good luck.