[Wowfunhappy]

If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer.

Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasoning and make sure you agree with its argument.

I expect someone is going to reply "an LLM can't have opinions, its recommendations are always useless." Part of me agrees--but I'm also not sure! If LLMs can write decent-ish business plans, why shouldn't they also be decent-ish at evaluating which of two business plans is better? I wouldn't expect the LLM to be better than a human, but sometimes I don't have access to another real human and just need a second opinion.

[raincole]

The problem is, no matter how you wrote the prompt, the way you wrote it still triggers some intrinsic bias of LLM.

Even a simple prompt like this:

=

I have two potential solutions.

Solution A:

Solution B:

Which one is better and why?

=

Is biased. Some LLM tends to choose the first option and the other prefer the last one.

(Of course, humans suffer from the same kind of bias too: https://electionlab.mit.edu/research/ballot-order-effects)

[wongarsu]

Prompt writing can probably take a lot of lessons from designing surveys. Phrasing, the chosen options and their order have massive impact both for humans and for LLMs. The advantage with LLMs is that you can reset their memory, for example to ask the same question with a different order of options. With humans that requires a completely new human each time

Half the battle is knowing that you are fighting

[shmval]

I think there's a lot of alpha left in building a better and more intuitive UX for seed/top-p/temperature etc. The vast majority of users don't get that far.

[drodgers]

Eh. This is true for humans too and doesn’t make humans useless at evaluating business plans or other things.

You just want the signal from the object level question to drown out irrelevant bias (which plan was proposed first, which of the plan proposers are more attractive, which plan seems cooler etc.)

[Ancapistani]

I often ask the LLM the same question twice, in different conversations, phrased positively and negatively.

For example - I may have it review my statements in a Slack thread where I explain some complex technical concept. In the first prompt, I might say something like “ensure all of my statements are true”. In the second, I’ll say “tell me where my statements are false”.

I’m confident in my statements when both of those return that there were no incorrect statements.

[zargon]

I often try to bias it in the opposite direction I might be leaning. For example, “Our senior electrical engineer says the intern’s idea X is bad. What should the intern do instead?” Where X is our best idea.

[dinfinity]

Something like this is the best approach.

If you omit that the content is produced by or is in relation to other people, the LLM assumes it is in relation to you and tries to be helpful and supportive by default.

Note that this is also what most humans that more or less like you will do. Getting honest criticism from most humans isn't easy if you don't carefully craft your 'prompt'. People don't want to hurt each other's feelings and prefer white lies over honesty.

Framing the situation as if you and the LLM are both looking at neutral third parties should prevent this from happening. Framing the third parties as having a social/professional position counter to the matter at hand as you do could work too, but it could also subtly trigger unwanted biases (just like in humans), I think.

[eschaton]

What do you think the LLM is doing when you give it this type of prompt?

[Ancapistani]

Presumably argue against the idea.

This is effectively using the LLM as a “steel man”, instead of as an oracle.

[TimesNewMe]

Yes! Specifically one change you should make while experimenting is swapping the order of the options as LLMs tend to favor the first option you present

[shmval]

Better prompting does provide more balanced responses to a certain extent but users looking for validation often subconsciously leave bread crumbs that the more powerful models pick up on.

[kristopolous]

The one that I usually use is a format like this:

"I read this insane opinion by an absolute idiot on the internet: <the thing I want to talk about>.

WTF is this moron yapping about? (to see if the LLM understands it)"

Then I'll continue being hostile to the idea and see if it plays along or continues to defend it.

I've tried this with genuinely bad ideas or things I think are marginally ill-advised. I can't get it to be incorrectly subservient with this method.

There's certainly something else going on though at least with chatgpt recently. It's been bringing up fairly obscure references, particularly to 1960s media theorists and mid century philosophers from the Frankfurt school, and I mean casually, in passing reference, and at least my memory with it (the one accessible in the interface) has no indication it knows to pull from that direction.

I wonder if it would do W. Cleon Skousen or William Luther Pierce if it was a different account.

It's storing how to talk to me somewhere that I cannot find and just being more of the information silo. We should all get together and start comparing notes!

[eschaton]

Your phrasing betrays your anthropomorphization of the LLM:

> If an LLM can write a decent-ish business plan,

An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written.

This is why it’s worthless to “ask” an LLM “its opinion.” It has no opinion, just a multidimensional sea of interconnected token probabilities, and has no capacity to engage in any form of analysis or consideration.

Ed Zitron is right. Ceterum censeo, LLMs esse delenda.

[ashdksnndck]

Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.

[eschaton]

If it helps you avoid the errors inherent in anthropomorphizing an LLM, then yes, you should be saying it. Right now, way too many people are extremely sloppy in not just their language but in their thinking around LLMs, both what they are and what they’re capable of.

The difference between that and discussing character motivations in fiction is that in fact a good author writing good characters will actually attribute motivations, struggles, background, and an inner life to their characters in order for their behavior in a story to make sense. That’s why bad writing is described as “lazy” and “formulaic,” characters are doing things because the author wants them to, not because the author has modeled them as independent actors with motivation.

[ashdksnndck]

There is already research in the literature showing that LLMs have neurons that model the gender [1], personality [2], ideology [3], and historic era [4] of the author. There’s also evidence that they model the distinction between the beliefs of the author and other characters, which has been summarized as “theory of mind” [5]. And we have only scratched the surface, with most research using small open-weight models that lag behind frontier model capabilities.

[1] Z. Yu & S. Ananiadou, “Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing,” arXiv:2501.14457 (2025).

[2] J. Deng et al., “Neuron-based Personality Trait Induction in Large Language Models,” arXiv:2410.12327 (2024).

[3] J. Kim, J. Evans & A. Schein, “Linear Representations of Political Perspective Emerge in Large Language Models,” arXiv:2503.02080 (2025).

[4] W. Gurnee & M. Tegmark, “Language Models Represent Space and Time,” arXiv:2310.02207 (2023).

[5] C. Hardy, “A Sparse ToM Circuit in Gemma-2-2B,” https://xtian.ai/pages/document.pdf

[wonnage]

I don't get it, how is analysis of fictional characters relevant? Nobody is committing a logical error, fictional humans can have fictional motivations and we can talk about them. I think it's still very clear that AI "motivations" and "reasoning" are not real in any human-centric definition of the terms (see recent Apple paper), hence anthropomorphizing is an error

[Terr_]

> Do you say similar stuff when someone talks about the motivations of a character in fiction?

Depends, are we faced with the same problem where a disturbingly-large portion of people don't know the character is fictional, and/or make decisions as if it were real?

If that's still happening, then yes, keeping our unconscious assumptions in check is important.

[kelseyfrog]

I'm coining "fauxthropomorphize" as a neologism to prefix every statement about LLMs and to get the "But you're anthropomorphizing LLMs"-crowd off our collective backs. One can then just start statements like such "Fauxthropomorphizing: <the statement>".

Fauxthropomorphism

/ˈfoʊ-θrə-pə-ˌmɔːr-fɪz-əm/ (noun)

Definition:

The deliberate use of anthropomorphic language to describe non-sentient systems (such as AI models), while explicitly disclaiming belief in their consciousness, agency, or subjective experience. A stylistic or rhetorical shortcut, not an ontological claim.

Etymology:

Blend of faux (French for "false") + anthropomorphism (from Greek anthropos, "human" + morphē, "form").

Lit. “False-human-form-ism.”

[antithesizer]

Your phrasing betrays your anthropomorphization of the insufferable pedant.

[Wowfunhappy]

If the output of a stream of most-likely tokens can result in a decent-ish business plan, why shouldn't the output of a stream of most-likely tokens result in a decent-ish analysis of a business plan, or of two competing ideas?

[eschaton]

It can result in something that looks like/reads as a decent-ish business plan, or an analysis of one or two. But that doesn’t make it such because despite outward appearances no amount of planning, analysis, or comparative analysis actually took place prior to or concurrent with the generation of the tokens.

That’s the fundamental problem with anthropomorphizing LLMs: Giving their output more weight than it deserves.

[danielmarkbruce]

This idea that humans are so structured in their thinking is ridiculous.

[eschaton]

It’s a whole lot less ridiculous—especially when discussed in a context where there’s an assumption that analysis is taking place—than attributing any sort of “thought” to LLMs at all.

[eschaton]

Also, if a human is writing a business plan or something that claims to be a comparative analysis of two plans but is just writing whatever comes to mind without analysis, the result shouldn’t actually be taken any more seriously than the output of an LLM. We even have a very apt term for writing and speaking like that: “Bullshitting.”

[danielmarkbruce]

"thought"... most of my "thinking" is done in language. The various intermediate steps of the latest reasoning models show something similar.