[nlawalker]

> no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information

"Be wowed by the convenience and speed", or merely "take advantage of the mere availability"? What most people find to be damning about expert advice is that they simply can't get it anywhere, at any cost that they can afford.

[whatever1]

So if you want to do a surgery but you don’t see any surgeons around you ask a grocery butcher to have his way?

[sxg]

In certain circumstances, the answer is yes. If an airplane's pilots are incapacitated, do you simply give up and crash the plane because there are no other pilots on board? Or would you rather have someone on the ground try to coach a passenger into at least attempting to land the plane?

[frereubu]

That's an extreme edge case, which I don't think is in the context of the concerns in this thread.

[sxg]

The specific case doesn't matter--it's meant to make you think about the general question throughout this thread: when an expert isn't available, should non-experts use AI (or other tools) to help themselves? Sometimes the answer is yes because the potential benefits outweigh the potential harms (if any harms exist). But sometimes the answer is no because misleading/incorrect advice can cause a net harm.

[frereubu]

But if the cases where AI use is a net positive are one in a million in medical situations? The argument is surely about the ratio, which many people here are arguing (from anecdote, would be interested to see a real study) is not in its favour, and the potential downsides - from both false positives and negatives - can be huge.

[ChrisMarshallNY]

As long as that passenger didn’t have the fish.

[close04]

A passenger crashing the plane while trying to avoid a certain crash doesn’t make things any worse. An incompetent doctor trying to save you from certain death can make things so much worse. It’s all about weighing the best/worst outcome compared to where you are now.

[microgpt]

I hate to break it to you but death is certain for everyone.

Properly emotionally processing this fact and your complete inability to do anything about it is called an "existential crisis" and if you haven't had one or several yet, you will.

[close04]

I’m not sure what the “revelation” is? How is this related to what I said?

Putting that aside, your philosophy sounds shallow. Death is certain, but how long you have to live and the quality of that life are not predefined. An incompetent passenger-pilot trying to save you from a crash will at worst make no difference. But an incompetent doctor can teach you that death isn’t necessarily the worst outcome.

[jancsika]

You can choose a) a calm, level-headed passenger who knows they aren't a pilot, or b) a calm, level-headed passenger who almost has their pilots license but has a medical condition that prevents them from admitting when they lack certain knowledge.

Who do you choose to be coached by an expert on the ground?

[rvnx]

No thank you, I will ask Claude and then ask ChatGPT to challenge me, and do a couple of rounds like that.

The first: Has no clue about anything and therefore no useful knowledge and cannot challenge me

The second one: Is proven to willfully give wrong information and will make me do mistakes for sure.

The LLMs will do their best, even if imperfect, since they summarizes what appeared in books.

I prefer to be grounded on what Airbus / Boeing manuals, or on what pilots training book said, than two far more unreliable sources.

[perching_aix]

No, people don't even go to a butcher, they do it themselves if they can. See the countless stories about farmers and their inventiveness. Example: https://www.youtube.com/watch?v=KKaJhQBusH8

[EA-3167]

People, especially in medical crises, are desperate for answers that they often can't get because their clinicians don't know. The illusion of an all-knowing guru who sounds like their doctor and tells them ANYTHING is extremely alluring. If you're waiting to hear back from a doctor about test results (which these days probably showed up on your online account the moment they were completed) can be agonizing.

Ok for pain in your shoulder it might not, but how about a woman with a lump in her breast waiting for the mammogram interpretation? How about someone trying to understand disturbing lab results? People are also often pushed these days to move through visits with doctors at a breakneck speed, but the AI will "hear you out" all day.

Part of this is a problem with the AI, part of it a problem with our healthcare systems, and part of it is simply human nature. If you think that OpenAI, Anthropic, Google and the rest weren't aware of this going in you must have very little faith in the intelligence of their members. It's not hard to imagine the future of LLM's should involve a hell of a lot of liability on the companies running it, but for now it's the Wild West.

[bilsbie]

> but how about

Whatever scenario you come up with my answer is the same.

As an adult I’d like to be able to choose what tools I use to learn about my condition regardless of how well it works or even if it’s likely to mislead me.

There’s risk in every aspect of life and we can’t baby proof everything.

[baconmania]

>choose what tools I use to learn about my condition regardless of how well it works or even if it’s likely to mislead me.

Even if it "works" so poorly that you're not actually learning about your condition?

[EA-3167]

If it's helping you learn about your condition then sure I agree. The issue here is that's not really the case, it's giving you the illusion that you're learning about your condition while feeding you hallucinations and half-truths at best. A recent look at medical advice from these things showed they're no better than a coin flip.

So if you MUST have answers that are at most random guesses, I'd suggest saving a few bucks and asking a coin before flipping it.

[perching_aix]

The companies are 100% aware, yes, and so they did make quite a few changes over the years.

Current trend is that the models will try to explicitly steer you towards "asking better questions from your medical provider", rather than providing diagnoses. They do also evaluate whether something can actually be established rather than just listen and nod along. And so the "you must have very little faith in the intelligence of their members" goes right back against these failure mode ideas.

Now of course, given a sufficiently desperate person, they can probably torture anything they want to hear out of these models. But so can they out of actual people, so that's kind of a high bar. When you get to the point where people are willfully misreading a given piece of text, bets tend to be rather off.