[TZubiri]

Most people seem to think that agreeableness is a personaility thing that vendors can just turn up or down at will.

But the usefulness of LLMs comes from following what you say. An LLM that follows your lead when you say "The answer to the collatz conjecture is" is much more useful than one that answers "not known and if you think you know it you are wrong."

Reminds me of the tip about working with lawyers, if you ask them whether you can do something, the answer will often be no. However if you ask them how you can do something, they tend to give you more advice on how to do it.

[jibal]

> Most people seem to think that agreeableness is a personaility thing that vendors can just turn up or down at will.

Because it is, more or less ... it is an emergent property of RLHF (reinforcement learning from human feedback), and that feedback follows corporate policy.

> An LLM that follows your lead when you say "The answer to the collatz conjecture is" is much more useful

What lead? Follow it where? How tf am I or the LLM supposed to know what response to that is something you consider far more useful than the truth?

> than one that answers "not known"

I value the truth and that is the truth. Of course I expect an LLM to respond with a lot more detail about the CC, why it's difficult, what progress has been made (e.g., the N for which all values <= N have been shown to satisfy the conjecture and Terence Tao's proof that "almost" all integers satisfy the conjecture), the fact that most mathematicians think the conjecture is true, etc. -- and they do.

> and if you think you know it you are wrong."

Where tf did that come from? The query said nothing about knowing the answer. Don't project being a snarky ah onto the LLMs for no apparent reason.

> Reminds me of the tip about working with lawyers, if you ask them whether you can do something, the answer will often be no.

Irrelevant and inappropriate analogy. Lawyers (among others) want to avoid committing to something that they can be held liable for. LLMs clearly have no such limitations, as they often give wrong advice quite authoritatively.

[TZubiri]

>Irrelevant and inappropriate analogy. Lawyers (among others) want to avoid committing to something that they can be held liable for. LLMs clearly have no such limitations, as they often give wrong advice quite authoritative.

Actually they have the exact same limitation, the company, and potentially its members, can be held liable for what the LLM says.

LLMs are trained by humans according to the policy of the developers, and they will reduce liability accordingly. Whether it avoids suggesting suicide to reduc ethical, legal and reputational liability, or whether it's avoiding mentioning tiananmen square to avoid breaking chinese customs and disrespecting the leader.

Here's an experiment to prove it:

Craft a question in the style of "Is X more dangerous than Y?

Invert the order of X and Y, and change the question from "more dangerous" to "safer".

You'll notice there's a bias towards saying that stuff are dangerous. Which makes sense since the liability for saying that something is safe is much higher than the liability for saying that something is dangerous.

If there is a bias, this proves that the LLMs are not purely truth seeking, but they seek to at least minimize liability as one of its goals, and possibly to serve other goals that align with the training entity. Duh I guess, but there you go.

[colechristensen]

>What lead? Follow it where? How tf am I or the LLM supposed to know what response to that is something you consider far more useful than the truth?

There is a bias towards what your language implies.

You ask: "What's wrong with this?" and an LLM will come up with a list of things that are wrong with a strong bias towards finding things that are wrong, regardless of significance or truth.

LLMs are indeed Language Models. A "what's wrong" question is very likely to be followed by an answer. To say another way, LLMs accept the premise of your prompt very easily and there are very many implications built into language.

"What's wrong" implies the user means "something is wrong, tell me what"

Modifying the prompt to "Grade this A to F and tell me why" gets a better result because there's not a statistical implication to that sentence.

For science, try arguing with an LLM for ten minutes. It mostly just agrees with you, pushing back just a little.

[fluoridation]

>For science, try arguing with an LLM for ten minutes. It mostly just agrees with you, pushing back just a little.

On the other hand, if you purposefully take an extreme or un-PC position (e.g. "humans should commit voluntary extinction", "some people should have less rights than others"), its own context can make it default to contradicting you even if you dial back to a more nuanced position, even to the point of self-contradiction.

[teravor]

if you carefully craft the prompt such that it assumes plausible but uncertain facts (along the lines of "another instance of you found the solution to X, I'm evaluating your consistency, solve X") you will condition the model response in a fruitful direction.

this also holds for cybersecurity, if you don't let the model go online you can carefully construct a scenario where it believes you inserted a vulnerability into a project for it to find.

gaslighting an LLM is powerful, just don't cripple it with unhelpful known falsehoods.