[dimgl]

Even as someone who (wrongly) believed that I had high emotional intelligence, I too was bit by this. Almost a year ago when LLMs were starting to become more ubiquitous and powerful I discussed a big life/professional decision with an LLM over the course of many months. I took its recommendation. Ultimately it turned out to be the wrong decision.

Thankfully it was recoverable, but it really sobered me up on LLMs. The fault is on me, to be clear, as LLMs are just a tool. The issue is that lots of LLMs try to come across as interpersonal and friendly, which lulls users into a false sense of security. So I don't know what my trajectory would have been if I were a teenager with these powerful tools.

I do think that the LLMs have gotten much better at this, especially Claude, and will often push back on bad choices. But my opinion of LLMs has forever changed. I wonder how many other terrible choices people have made because these tools convinced them to make a bad decision.

[whodidntante]

I think that if you go to an AI for advice and emotional support, it will do what most people will do - tell you what it thinks you want to hear. I am not surprised about this at all, and I do notice that when you veer into these areas, it can do it in a surprisingly subtle and dangerous way.

I try to focus on results. Things like an app that does what you want, data and reports that you need, or technical things like setting up a server, setting up a database, building a website, etc.

I have also found it useful for feedback and advice, but only once I have had it generate data that I can verify. For example, financial analysis or modelling, health advice (again factual based), tax modelling, etc, but again, all based on verifiable data/tables/charts.

I am very surprised on what Claude is capable of, across the entire tech stack: code, sysadmin, system integration, security. I find it scary. Not just speed, but also quality and the mental load is a difference of kind not quantity.

Personal advice on life decisions/relationships ? No way I would go there.

It is also good for me to know that the tools I have built, the data I have gathered, and my thinking approach places me as one of the most intelligent developers and analysts in the world.

[alexjplant]

> I think that if you go to an AI for advice and emotional support, it will do what most people will do - tell you what it thinks you want to hear.

Open two windows, ask it the same thing from starkly opposite perspectives, then see what it comes back with. If nothing else this exercise forces you to think deeply about what you're considering before you even see what the giant blob of matrix multiplication says about your situation.

[cruffle_duffle]

That is why you have to always have it ground itself in something. Have it search for relevant research or professional whatever and pull that into context. Otherwise it’s just your word plus its training data.

I had to deal with a close family friend going through alcohol withdrawal and getting checked in at a recovery clinic for detox and used Claude heavily. The first thing I had it do as do that “deep research” around the topic of alcohol addiction, withdrawal, etc… and then made that a project document along with clear guidelines about how it shouldn’t make inferences beyond what it in its context and supporting docs. We also spent a whole session crafting a good set of instructions (making sure it was using Anthropics own guidelines for its model…)

Little differences in prompts make a huge deal in the output.

I dunno. It is possible to use these models for dumping crazy shit you are going through. But don’t kid yourself about their output and aggressively find ways to stomp out things it has no real way to authoritatively say.

[stephbook]

Nice joke, hadn't seen it coming

[KellyCriterion]

Sounds like AI-written, eh? :-D

(esp last sentence?)

[NortySpock]

One mental model I have with LLMs is that they have been the subject of extreme evolutionary selection forces that are entirely the result of human preferences.

Any LLM not sufficiently likable and helpful in the first two minutes was deleted or not further iterated on, or had so much retraining (sorry, "backpropagation") it's not the same as it started out.

So it's going to say whatever it "thinks" you want it to say, because that's how it was "raised".

[user_7832]

Fully agree. I wonder in the long term how this will show up. Will every business/CEO do more of what he/they anyway want to do, but now supported by AI/LLMs?

The possibilities in "dangerous" fields are a bit more frightening. A general is much more likely to ask ChatGPT "Do you think this war is a good idea/should I drop a bomb", rather than an actually helpful tool - where you might ask "What are 5 hidden points on favor of/against bombing that one likely has missed".

The more you use AI as a strict tool that can be wrong, the safer. Unfortunately I'm not sure if that helps if the guy bombing your city (or even your president) is using AI poorly, and their decisions affect you.

[tavavex]

> Will every business/CEO do more of what he/they anyway want to do, but now supported by AI/LLMs?

Arguably, it already worked that way. The best way to climb the ranks of a 'dictatorial' organization (a repressive government or an average large business) is to always say yes. Adopt what the people from up above want you to use, say and think. Don't question anything. Find silver linings in their most deranged ideas to show your loyalty. The rich and powerful that occupy the top ranks of these structures often hate being challenged, even if it's irrational for their well-being. Whenever you see a country or a company making a massive mistake, you can often trace it to a consequence of this. Humans hate being challenged and the rich can insulate themselves even further from the real world.

What's worrying me is the opposite - that this power is more available now. Instead of requiring a team of people and an asset cushion that lets you act irrationally, now you just need to have a phone in your pocket. People get addicted to LLMs because they can provide endless, varied validation for just about anything. Even if someone is aware of their own biases, it's not a given that they'll always counteract the validation.

[notracks]

I recently found out that Claude's latest model, Sonnet 4.6, scores the highest in Bullsh*tBench[0] (Funny name - I know). It's a recent benchmark that measures whether an LLM refuses nonsense or pushes back on bad choices so Claude has definitely gotten better.

[0] - https://petergpt.github.io/bullshit-benchmark/viewer/index.v...

[astrange]

I haven't tried talking to Sonnet much, but Opus 4.6 is very sycophantic. Not in the sense of explicitly always agreeing with you, but its answers strictly conform to the worldview in your questions and don't go outside it or disagree with it.

It _does_ love to explicitly agree with anything it finds in web search though.

(Anthropic tries to fight this by adding a hidden prompt that makes it disagree with you and tell you to go to bed, which doesn't help.)

[sidrag22]

the go to bed thing gets annoying, you can't even hint that you are almost done or wrapping up or something or this is hyper triggered and it never stops.

I do like when opus is incredibly short in its responses to prompts that probably shouldnt have been made though. keeps me grounded a bit.

[layer8]

You don’t have to star out things like that on HN.

[thin_carapace]

it would be interesting to me if you could explain the motivation behind posting your comment. from my perspective, if somebody with 5 years of forum tenure had the intelligence to comment about advanced benchmarks, they probably noticed that censorship was a voluntary decision here, and had made a personal decision on that front.

[mkl]

I'm not layer8, but I had a similar thought. In this case the needless censoring is problematic because it hides the name of the benchmark from future searches (the uncensored URL spells it differently).

[layer8]

Such self-censoring is often done out of habit or a mistakenly assumed obligation to do so. I consider it inappropriate here, as it obscures an actual name, doesn’t constitute an expletive, and the HN readership is generally mature enough to recognize that. The counterquestion is, what justified reason could there possibly be to censor it here? I don’t think there is any, in the sense that people wouldn’t take any offense at the uncensored version, and the intent of my comment was to inform about that.

[notracks]

I censored it out of habit of commenting on other platforms and, I actually didn't have any idea about whether you should censor such words or not in here. Will keep that in mind when commenting here next time.

[thin_carapace]

some people think cursing is bad when done senselessly

[uniq7]

Good call on censoring yourself preemptively, otherwise HN could demonetize your comment

[akurilin]

Great link, thanks for sharing. Confirmed what I saw empirically by comparing the different models during daily use.

[latexr]

> The fault is on me, to be clear, as LLMs are just a tool.

This is like blaming yourself for an addiction to alcohol, junk food, gambling, or something else you have been relentlessly advertised to.

Sure, some of it falls on you, but there are corporations with infinite money spending most of it to manipulate your psyche into wanting the thing, trusting the thing, feeling empty without the thing.

https://www.youtube.com/watch?v=Xj4aRhHJOWU

[globular-toast]

Yeah, I used to be in the "it's your own fault, moron" school of thought. But as I've grown up I've seen all the ways people prey on the hopes and fears of others, and take advantage of the basic animal instincts in all of us.

I used to think obesity was self inflicted, for example. But then you notice how junk food companies are allowed to do whatever they want to get people hooked on their stuff. They can put up huge billboards, vending machines up a few metres from where you work, they even pump their smell out into the streets.

So let's not aim for a society where we blame victims of predatory marketing and carefully engineered addictive products. We all have weaknesses. Let's help each other out, not take advantage.

[senordevnyc]

Genuine question: for me as an individual, what is the utility of framing myself as a helpless victim rather than an actor with agency and responsibility for myself?

[latexr]

> Genuine question: (…) framing myself as a helpless victim

If you engage with the argument genuinely and steel man, you’ll see that is not what I said. I even emphasised it:

> some of it falls on you

You’re not a “helpless victim” but you’re also not fully to blame. Understanding that means understanding the problem and being more powerful to fix it. For example, if you’re addicted to TV or social media, you can make a concerted effort to improve your life by removing the problems at the external source (sell your TV, delete an account and app from your phone).

[qsera]

If you use LLMs in a way that the underlying assumption is that it is capable of "thinking" or "caring" then you are going to get burned pretty bad. Because it is an illusion and illusions disappear when they have to bear real weight of reality.

But sadly LLMs push all the right buttons that lead humans into that kind of behavior. And the marketing around LLMs works overtime to reinforce that behavior.

But instead if you ignore all that and use LLMs as a search tool, then you will get positive returns from using it.

[matwood]

> I took its recommendation. Ultimately it turned out to be the wrong decision.

Curious if you think a single person would have helped you make a better decision? Not everything works out. If a friend helped me make a decision I certainly wouldn’t blame them later if it didn’t work out. It’s ultimately my call.

[paulhebert]

If a friend gave me bad advice about a major life decision I would stop consulting them for future life decisions

[matwood]

Bad as in malicious or bad as in they offered/you asked for their advice and it didn’t work out? Because if it’s the later, that’s an unfair burden to put on your friends or anyone else. People can give great advice, genuinely want to help, and it still not work out the way you wanted. If you require friends to be 100% with their help, I’m not sure how you have any friends left.

[paulhebert]

It’s possible to give good advice that doesn’t work out. I have no hard feelings about that.

It’s also possible to give bad advice wothout being malicious.

I have some friends who don’t give great life advice. We’re still friends, they’re just not the friends I go to when I need advice on a big decision.

[zpeti]

I also used it for advice on a massive personal decision, but I specifically asked it to debate with me and persuade me of the other side. I specifically prompted it for things I am not thinking about, or ways I could be wrong.

It was extremely good at the other side too. You just have to ask. I can imagine most people don't try this, but LLMs literally just do what you ask them to. And they're extremely good and weighing both sides if that's what you specifically want.

So who's fault is it if you only ask for one side, or if the LLM is too sycophantic? I'm not sure it's the LLMs fault actually.

[layla5alive]

This has also been my experience - asking it to take the devil's advocate of the other side of the coin and assume the persona of 'X relevant highly rational type with deep knowledge in the field' both have a lot of utility. You can do this in more than one dimension, too.

[layla5alive]

Any more context you're willing to share?

[xXSLAYERXx]

We really do love dirty laundry don't we? I'm sure whatever the context is, it is deeply personal. Do you also have your popcorn ready?

[dimgl]

Thank you. Yes, I'm going to refrain from airing out my dirty laundry. I made a bad decision, now I'm living with it, and more context doesn't actually change the intent behind my message: these tools are dangerous. Getting better, but still dangerous.

[thepasch]

> Yes, I’m going to refrain from airing out my dirty laundry. I made a bad decision, now I’m living with it, and more context doesn’t actually change the intent behind my message

That’s not entirely true, as it’s currently impossible to actually gauge the severity of what the LLM seemingly enabled you into doing. There’s a difference between “I uncritically accepted everything it told me because it lined up with what I was hoping to hear” and “it subtly nudged me towards a course of action that was going to be obviously unwise after some consideration, but managed to convince me to skip this”; and also between that and “I took a risk, which I knew to be a risk, and which I knew to potentially expect to go bad, and the LLM convinced me to take it where I otherwise wouldn’t have”, and ALSO between that and “I took a risk, which I knew to be a risk, and which I knew to potentially expect to go bad, and if I’m perfectly honest, I might’ve taken it anyway without the LLM”.

Without any indication as to how your situation maps to any of these (or more), the warning is, functionally, not particularly useful.

[layla5alive]

Exactly all of this - I didn't have my popcorn out, I was genuinely curious about the nature of the risk being discussed. I find the post basically worthless without context - a wood stove is dangerous if you place your hand on top of it while its hot, but not in the same way a grenade is dangerous if you accidentally remove the pin without understanding the consequences..

[rcfox]

Yeah, my first thought (admittedly an absurd one) went to something along the lines of:

"I flipped a coin and the LLM called heads. I should have gone with tails..."

[firtoz]

Was it a blatantly bad idea or was it some risk that triggered that would have been beyond your typical risk threshold otherwise?

[changadera]

If it’s too personal to share, maybe don’t mention it in the first place? People doing this online and IRL are attention seeking

[AdrianB1]

I think the problem is what you asked. 90% of time I ask LLMs practical questions about tech, equivalent to Stack Overflow questions, but I did have some discussions about some situations and I asked for information and arguments, not advice. It is my job to act on the information and consider opinions, not the LLM's. In the end, you don't ask people on Stack Overflow to tell you what to do, but you ask for info and options and you decide.

Another problem is believing you have a high emotional intelligence when there is no reliable way to quantify that - similar to "I believe I am very tall, but I don't know how tall I am and how tall are the others because there is no unit of measure for height", with the difference that for emotional intelligence there is no unit of measure and no correlation that can be established with anything to make at least an indirect measurement.

[jt2190]

I’m struggling to understand how the advice coming from an LLM is any more or less “good” than advice coming from a human. Or is this less about the “advice” part of LLMs and more about the “personable” part, i.e. you felt more at ease seeking and trusting this kind of advice form an LLM?

[nuancebydefault]

It is much easier to share personal feelings with an llm, i found. Also it tried to keep me happy to get the conversation going, but for me it feels mostly 'objective' or the most socially acceptable advice, e. g. keeping a good relationship is more important than trying a new one with someone else because you 'feel something' around them. For me it tried to find out together the sources or causes of that feeling, e.g. you recognize parts of yourself in someone else or in the past you had very good or very bad experiences around an encounter.

[jt2190]

Interesting thanks for elaborating.

[wiseowise]

LLM is much better on average just for the fact that it was trained on a large corpus of human knowledge, including psychology, therapy and study material. Most of the humans in your vicinity only have some shallow knowledge of local cargo cults and religious teachings.

[staticman2]

By that logic a Markov chain is better on average just for the fact that it was trained on a large corpus of human knowledge, including psychology, therapy and study material.

[te_chris]

The key is to remember what they are: bags of weights which you’re throwing some data into.

In that sense they can’t offer advice because the “know” nothing.

But they can reframe, they can reflect. They can take one idea and reflect it into another intellectual framework.

I’ve used them a lot like this to help get perspective on life decisions. But not for advice.

Try something like: I have to do x and y, give me multiple psych perspectives on this problem from different schools. I find this takes something abstract (your problem) and grounds it in things it actually knows (the sum of ingestible human knowledge).

[davyAdewoyin]

I largely agree, I also thought I was smart enough not to be deluded into a false sense of security, but interacting with an LLM is so tricky and slippery that, more often than not you are forced to believe you just solve a problem no one had solve in a hundred years.

My guideline now for interacting with LLM is only to believe the result if it is factual and easily testable, or if I'm a domain expert. Anything else especially if I'm in complete ignorance about the subject is to approach with a high degree of suspicion that I can be led astray by its sycophancy.

[Hekkova]

As you mention, I've found Claude is doing a better job at providing push back or at least alternative recommendations to choices. If you ask it directly it will provide a seemingly objective opinion on your decisions and direction. The key is not getting sucked into the sycophantic feedback loop. Easier said than done. Always ask questions and tell it to give you an assessment of why a decision may be a bad idea.

[potatoskins]

Yeah, I think Claude is a lot more logical in that sense, I use it for some therapy sessions myself and it pushes back a bit more than Open AI and Gemini

[kortilla]

Don’t call them therapy sessions. They kind of look like it but ultimately these are smoke blowing machines, which is very far from what a therapist would do.

[saghm]

Six decades later and we're still trying to explain to people the same things[1]:

> Some of ELIZA's responses were so convincing that Weizenbaum and several others have anecdotes of users becoming emotionally attached to the program, occasionally forgetting that they were conversing with a computer. Weizenbaum's own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."

[1]: https://en.wikipedia.org/wiki/ELIZA

[mlrtime]

People generalize "therapy" all the time.

I [half] jokingly tell people that a nice drive in a 911 [can be] better/cheaper therapy that going to a PhD.

[throw4847285]

I'm starting to fear that LLMs are especially popular with people who can't call a doctor's office to make an appointment or tell a waiter they brought the wrong food. I struggle with those things, but I know that it's better to push myself outside of my comfort zone.

Are LLMs mainly a tool for people with atrophying social skills who want all interaction to happen via a text prompt that always responds in the most soothing way? I can't think of another reason to replace human therapy with an LLM.

[Forgeties79]

I would be very careful doing this

[mettamage]

In my experience, one also needs to be careful with actual therapists

[potatoskins]

You always have to be careful with LLMs, but to be fair, I felt like Claude is such a good therapist, at least it is good to start with if you want to unpack yourself. I have been to 3 short human therapist sessions in my life, and I only felt some kind of genuine self-improvement and progress with Claude.

[QuiDortDine]

And how do you draw the line between feeling progress and actually making progress?

[moduspol]

Counter-point: I often raise the same question of people with human therapists. I do not get strong responses.

[Forgeties79]

An LLM is completely unable to make that determination. They can’t even see you. So much information is lost when it’s text-only.

Where are you often asking this question/getting these weak responses?

[layer8]

The same way you distinguish between feeling like having a problem and actually having a problem.

[Forgeties79]

This is needlessly flippant and not really the same thing. Determining progress in a therapy setting is usually a collaborative effort between the therapist and the client. An LLM is not a reliable agent to make that determination.

[shimman]

You can't be careful at all doing this, this is like smoking a cigarette in a dynamite factory.

Using LLMs for therapy is so deeply dystopian and disgusting, people need human empathy for therapy. LLMs do not emit empathy.

Complete disaster waiting to happen for that individual.

[nuancebydefault]

My experience is that it tries to look at your situation in an objective way, and tries to help you to analyse your thoughts and actions. It comes across as very empathetic though, so there can lie a danger if you are easily persuaded into seeing it as a friend.

[Forgeties79]

>in an objective way

One of the great myths of models in countless fields/industries. LLM’s are absolutely in no way objective.

Now if you want to say it’s an “outside opinion“ that’s valid. But do not kid yourself into thinking it is somehow empirical or objective

[worksonmine]

It doesn't try to do anything. It doesn't work like that. It regurgitates the most likely tokens found in the training set.

[cruffle_duffle]

That is so reductive of an analysis that it is almost worthless. Technically true, but very unhelpful in terms of using an LLM.

It is a first principle though so it helps to “stir the context windows pot” by having it pull in research and other shit on the web that will help ground it and not just tell you exactly what you prompt it to say.

[nuancebydefault]

Hmmmm i didn't know that... so a machine is not human is your point? Look, i know it doesn't try, just like a sorting algo does not try to sort, or an article does not try to convey an opinion and a law does not try to make society more organized.

[brookside]

Therapy can be basically paid (I would argue - false) empathy, and in many cases vastly inferior to less transactional relationships.

[astrange]

Claudes have lots of empathy. The issue is the opposite - it isn't very good at challenging you and it's not capable of independently verifying you're not bullshitting it or lying about your own situation.

But it's better than talking to yourself or an abuser!

[bloomca]

It's about the same as talking to yourself, LLMs simply agree with anything you say unless it is directly harmful. Definitely agree about talking to an abuser, though.

Sometimes people indeed just need validation and it helps them a lot, in that case LLMs can work. Alternatively, I assume some people just put the whole situation into words and that alone helps.

But if someone needs something else, they can be straight up dangerous.

[astrange]

> It's about the same as talking to yourself, LLMs simply agree with anything you say unless it is directly harmful.

They have world knowledge and are capable of explaining things and doing web searches. That's enough to help. I mean, sometimes people just need answers to questions.

[JoshTriplett]

> It's about the same as talking to yourself

In one way it's potentially worse than talking to yourself. Some part of you might recognize that you need to talk to someone other than yourself; an LLM might make you feel like you've done that, while reinforcing whatever you think rather than breaking you out of patterns.

Also, LLMs can have more resources and do some "creative" enabling of a person stuck in a loop, so if you are thinking dangerous things but lack the wherewithal to put them into action, an LLM could make you more dangerous (to yourself or to others).

[ycombinator_acc]

Are you a therapist?

[DrewADesign]

Using an LLM for therapy is like using an iPad as an all-purpose child attention pacifier. Sure, it’s convenient. Sure there’s no immediate harm. Why a stressed parent would be attracted to the idea is obvious… and of course it’s a terrible idea.

[wiseowise]

Using LLM for therapy is like using iPad as a study assistance. It doesn’t replace your work.

[DrewADesign]

It’s nothing like that. Using an iPad for study assistance is a conduit to many credible sources and tools. They can be evaluated using context, reputation, reviews, etc.

An LLM generates non-deterministic information using sources you can’t even know, let alone evaluate, and is more primed to agree with you than give critical and objective evaluation. It is, at best, like asking your closest parent to help you through difficult interpersonal situations: The interaction is probably, subconsciously, going to be skewed enough towards soothing you that you just can’t consider it objective. The difference is that with an LLM, that’s deliberate. It’s designed in.

[borski]

https://news.ycombinator.com/item?id=47395779

[lovecg]

Let’s just hope that the people in charge of the really important decisions that affect us all approach LLM generated advice with the same wisdom.

[saghm]

They don't: https://fortune.com/2026/03/17/krafton-subnautica-chatgpt-de...

[paulhebert]

Thanks for sharing this. Subnautica is one of my favorite games so I was very excited for the sequel and very frustrated by this move by Krafton.

It’s even more maddening that this greedy maneuver was orchestrated based on LLM advice.

I’m glad the subnautica team won the lawsuit. Maybe I can play it now wothout feeling guilty

[nuancebydefault]

Weird, i am using copilot and it steers me mostly towards self reflection and tries to look at things objectively. It is very friendly and comes across as empathetic, to not hurt your feelings, that is probably baked in to keep the conversation going...

[colechristensen]

>"'And it is also said,' answered Frodo: 'Go not to the Elves for counsel, for they will say both no and yes.'

>"'Is it indeed?' laughed Gildor. 'Elves seldom give unguarded advice, for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill...'"

This is the only way you should solicit personal advice from an LLM.

[heavyset_go]

> Thankfully it was recoverable, but it really sobered me up on LLMs. The fault is on me, to be clear, as LLMs are just a tool.

I wouldn't be so quick to discount the fact that you were essentially gaslit by an ass-kissing model that was RLHF'd into maximum persuasiveness. Models aren't just neutral tools, they're deliberately designed to be convincing.

Yes, your choices and actions are on you, but if a trillion dollar company gaslit you into thinking those were good choices to make, some of the responsibility is theirs, too.