[dirkt]

What happened to VERIFYING an answer? Does nobody do that anymore?

When I ask an LLM, I trace the sources, and see if they make sense.

More often than not the sources don't actually say anything about the topic in particular...

> It’s also scary how easily you can lead each LLM to the answer you have in mind.

Exactly. Which is why "treat an LLM like a human expert who can answer your question" doesn't work. It's more like a human bullshitter who makes up convincing looking answers, and tries to please you. If the answers have actually some grounding in the training material, that's useful as some kind of holistic google, but often it's not.

[palata]

> What happened to VERIFYING an answer? Does nobody do that anymore?

The problem with medical advice is that you may not be competent to verify the answer, right?

I agree that asking 5 LLMs to vote and trusting the answer is totally the wrong approach, of course. But LLMs (and traditional material) can help getting more informed. For instance, instead of going to your doctor with the LLM diagnosis and trying to convince the doctor that the LLM is right, you can try to build your own understanding of the problem and go ask the doctor to explain to you what you understood correctly and what you misunderstood.

If you have some understanding, it's harder for a specialist to bullshit you. But you need your own critical thinking and you need to put effort into actually learning something, blindly trusting and repeating what LLMs say doesn't help.

[spwa4]

Or more specifically in this case: the patient was obviously insisting on a diagnosis and treatment based on ... a slightly hurting shoulder, with zero visible or detectable phenomena.

So the doctors gave him what he wanted: a treatment ... and Claude told him the treatment was a placebo. Correctly, I might add.

Yeah, it is absolutely not what the patient wanted to hear. BAD doctors! Except ... no, not really.

Does that explain what happened here?

[palata]

Hmm that's not what I read from the article. The author says the opposite, actually:

> [the orthopedist] suggested I get an MRI, which the clinic conveniently had available. [...] This, of course, means little to me, but their suggested course of treatment was extensive; [...] Coming out of the clinic, I had the feeling they had jumped the gun.

The author also says:

> They injected me with Traumeel, which is registered in Germany as a homeopathic medicine "without a therapeutic indication".

I personally wouldn't want to be injected homeopathic medicine "without a therapeutic indication" without even knowing it is homeopathy.

And my recent experience with multiple doctors at multiple hospitals is that just like LLMs, you shouldn't blindly trust them. Sometimes they make mistakes, and in my experience they never, ever admit it (maybe even to themselves).

So trying to get informed "on the internet" (including with LLMs) feels sane to me, but that's worth what it is worth.

What concerns me in the article is that they have GPT make a diagnosis, Claude review it, and somehow seem to assume that if both LLMs agree, they must not be completely wrong. Just like for code, it takes an expert to leverage an LLM to take an expert decision. A beginner can leverage the LLM to understand the problem better, but they never reach the level of expert just from that.

Unfortunately it's part of lowering the confidence towards doctors, and the solution to that is to try and get informed and ask another doctor. And of course they don't like it if you say "I already asked someone else but I don't trust doctors in general, so I am now asking you to test the both of you".

[capr]

so the doctors acted just like the LLM

[spwa4]

Quite the opposite. The doctors are disagreeing with the LLM, and when the patient doesn't accept, they're putting in a minimal lie in the deontologically accepted way.

This seems to be to be near the opposite of what a sycophant (ie. an LLM) would do.

[prmph]

I've also noticed the opposite problem: Sometimes the LLM, when asked a detailed question (probably with some lead-in), pushes back in a way that betrays that they fell back to general tropes without really considering the nuances of your specific context.

This happens many times, and I usually have to lead the LLM through a chain of reasoning to prove to it that its objection, through generally sound, do not apply to my specific situation.

Someone not as well versed in the subject matter would think the LLM found a smoking gun (which they love to do), and be led on a wild goose chase.

[wizzwizz4]

> I usually have to lead the LLM through a chain of reasoning to prove to it

What's the point of doing this?

[prmph]

So that hopefully we can go further in the discussion without it having to repeatedly bring up those (discredited) objections.

But it does forget, and I'd have to prime it again for another session.

[mathieuh]

As you say, often you check up on the LLM's "reasoning" and it doesn't follow at all, or you can easily get it to contradict itself with just as much certainty as it had about its previous convictions.

It is very scary to me that people are entrusting potentially life-altering decisions to these things.

[otabdeveloper4]

> When I ask an LLM, I trace the sources, and see if they make sense.

Professional tip: you can cut out the LLM middleman here and save a lot of time and money.

[microgpt]

What would you use then? Google Search, which is just a shittier LLM?