[rafterydj]

I feel like I'm going nuts.

There are other commenters saying this is a good practice they've also done for other injuries. You are saying you are an actual radiologist and immediately clock the problems with its advice.

I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading. It is only when you do not know what the AI is being asked to do is it likely you will find the output helpful.

This is itself alarming to me, but no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information.

[appplication]

This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief.

It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.

Don’t get me wrong, I think we all agree capabilities will eventually improve (and farther-future capabilities could reasonably surpass experts), but really is unclear if the current transformer architectures with their probabilistic/hallucinatory outputs will plateau before they surpass current experts abilities in all promised fields.

[cheschire]

I was a very early adopter in my circles with AI and I shared it with many people. Strangely, I seem to be the most skeptical about AI in my circles as well, but because I was the gateway for a many folks, they want to come back and share their experiences with me.

And it's so much like listening to someone in a church congregation sharing their experiences with god. Clear and obvious gaps are hand-waved away exactly how you're describing.

[operatingthetan]

>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem.

The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.

[aetherson]

You're confusing the training method with the internal process. If I had you repeatedly attempt to learn how to make believable completions of partial documents about a given topic, you would eventually learn things about that topic and could use your knowledge to create more believable completions of documents about that topic.

[goodpoint]

believable != true

[operatingthetan]

LLMs do not learn. You put it out to pasture and create a new one. "Memory" in a session is essentially a context window party trick.

[lemiffe]

The LLM itself doesn't, but agents can research, compare, add to their memory, and use that to narrow the results down to a probabilistically higher set of outputs; I have used an LLM for my own MRI results and it was nearly spot-on, verified by a subsequent visit to a specialist. YMMV as they say. But I do believe we are entering the era where LLMs are considering past interactions and long context windows to inform it of personal preferences and history in order to output more accurate results.

[aspenmartin]

They do learn in context, and very sample efficiently. Continual learning is active area of research and we sort of already have something resembling it with persistent context. So yes they do learn.

[operatingthetan]

I consider that to be the illusion of learning. You are not wrong, I think they may actually learn in the future though. But not today.

[singpolyma3]

That presumes that we have a definition of "thinking" or that we know that anything is "thinking" when in fact neither is true.

The problem is real but I don't think positing a philosophical root is helpful

[operatingthetan]

The claim that we are assigning human-like agency to a machine with none is simple and factual.

[ForceBru]

What's "thinking"? What's "agency"? What's "human-like agency"?

If "agency" is making decisions and performing corresponding actions in the real world, then LLMs most definitely LOOK LIKE they're making decisions (what's the next token? which tool to use? what's to say, in general? what idea to convey?) and performing actions (tool use). Can we tell whether they are ACTUALLY making decisions? Well, are the people around me "actually" making decisions? Or are they simply pushed around by circumstances and external forces?

Am I actually making decisions? Did I like DECIDE to write this comment? Maybe? I have no clue...

[operatingthetan]

I think you're mildly obfuscating the issues at hand by diving too deeply into philosophical questions.

It's quite simple, the agency that the LLM appears to have is actually your own. Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems.

[singpolyma3]

The idea that humans have agency is supernatural thinking imo

[operatingthetan]

A free will versus determinism argument doesn't really have a place here. Consider instead that humans factually have 'the illusion of agency.' The LLM does not even that have that. It cannot act on it's own, it has no ongoing drama or intention. It only reacts to prompts.

[corndoge]

Often times the words produced do have legitimate factual information though. It's less psychosis and more a confluence of well known human tendencies - salience bias, automation bias, etc.

[operatingthetan]

The big problem is often times they don't as well. That's why we can't rely on them.

[aspenmartin]

Same with humans? Doctors, scientists...if a tool has any error rate above zero its not reliable?

[TomasBM]

Why is it psychosis and not lower standards?

While I can understand being skeptical of non-experts' claims that such answers are enough, I don't understand why you call it "psychosis" and not simply naivety or lack of expertise.

At the same time, the new so-called "models" haven't been pure transformer-based LLMs, but entire systems with tools (with access to the Internet), data storage, and the options to trigger additional instances for different tasks.

[janmatejka]

Because some people develop actual psychosis. They go down some rabbit hole with an LLM until the LLM makes them believe they invented new kind of physics that makes them go harassing experts who obviously try to ignore them because its all nonsense.

[perching_aix]

[delayed]

[ruszki]

For me, what others said and literally showed with Claude Code, et al, and what I’ve been experiencing with it, clearly signal way lower standards. But this was true even before LLMs.

[shimman]

Reminds me of that clip of Travis Kalanick, sexual deviant and harasser of women, talking about "discovering new physics."

[natebc]

The Uber guy? Yeah that was a painful watch.

[lazide]

I don’t think they will improve, there is too much incentive to poison the datasets going forward.

A lot of the models up to this point have been benefitted - like Google did - from essentially ‘pre SEO’ internet.

Now the same tools are being used to generate nigh infinite good sounding bullshit, which poisons the dataset in all sorts of hard to detect ways.

To add insult to injury, the human experts are also not as. Naive, and have many incentives to poison their own input in subtle ways too.

[brokencode]

I seriously doubt that data set poisoning will be a real limiter in model performance.

For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models?

For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc.

Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.

[chmod775]

There's a post every other month where some dude who put nonsense information online celebrates because it actually ended up in some frontier models weights.

If it's easy enough that some randos can do it for fun, what do you think happens when there's commercial interest behind it?

Obviously companies are going try nudging AI towards recommending whatever they're selling. It's a logical extension of SEO - and that's a 100 billion USD industry.

Additionally, if I believed myself to be in some sort of spending - err - AI race, I'd try to poison the data sets of my competitors by putting crap out there for others to ingest.

[aspenmartin]

It's not really a problem. We're out of natural tokens anyway. The future is synthetic verifiable traces (already the way we train coding agents).

[jurgenaut23]

Do you have examples of such celebrations?

[Analemma_]

I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.

[aspenmartin]

Data curation is important and expensive and frontier labs can afford to do it right. Natural data isn't the limitation, we are already literally out of tokens. It doesn't matter how much you poison things it's not going to stop the progress train.

[tayo42]

Who's doing llm seo right now? How does that work when you only gets feedback every few months when a new model is out?

[natebc]

I'm pretty sure the Optimization part is just ... not present at all.

This is how we get LLM summaries presenting something mentioned once by some nutjob in a reddit thread as bona fide FACT

[DougN7]

Look at G2.com - they found their website is highly references by AIs and they are leaning into it hard.

[microgpt]

Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else.

Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates.

Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.

[thfuran]

>Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates.

It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.

[rvnx]

Human doctors use LLMs to diagnose too

OpenEvidence claims

    "More than 40% of U.S. physicians use it daily, and it handled around 20 million clinical consultations per month. Over 100 million Americans were treated by a doctor using it in 2025."

https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doc...

[something98]

This is a very misleading statement; most of those physicians are using LLMs to transcribe notes from visits and/or for billing purposes (e.g., proper billing codes).

[kjellsbells]

The problems isnt LLMs per se, it is the shift to trusting the output of the machine coupled with a decline in verifying that the output is reasonable. It's basically what your teachers warned you about with wikipedia in eight grade except applied to all areas of life, including medicine. Dictation is already high-stakes and LLMs do not automatically reduce that risk.

Here is an example. My provider sent me this note. I'm quoting verbatim here from my MyChart record:

"Your liver enzymes are high, I would like to order acetaminophen containing medication like Tylenol, I would like to order liver ultrasound I placed ultrasound order in the system, make an appointment for radiology, I would like you to get hepatitis panel lab work done, obtain blood work order, please schedule a well visit to get it done"

When I queried it, this is what I got back. It was a dictation error. You could almost hear the panic in the message:

"Sorry for wrong message earlier, I was dictated message- so could not realize that it was written to take Tylenol type of medicines- I DO NOT RECOMMEND ACETAMINOPHEN CONTAINING MEDICINE - LIKE TYLENOL AND ALCOHOL DUE TO ELEVATED LIVER ENZYMES."

Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.

[ethbr1]

> Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.

100%. Also, management.

I wish someone would go ahead and coin an AI version of Amdahl's law that states the work speedup from AI is dependent on amount of unverified AI output used.

Iow, if you 1:1 verified everything, there would be no time savings.

Ergo, you get management saying (1) we demand time savings due to AI & (2) we demand you fully check anything you use AI for.

End result? People skip (2) to hit (1).

Then management burns anyone at the stake whenever inevitable mistakes happen.

[brokencode]

OpenEvidence is specifically meant to help clinicians make evidence-based decisions in the diagnosis and treatment of patients, not note transcription.

[sxg]

It does both: https://www.openevidence.com/user-guide/visits-overview

[sarchertech]

Ignoring the fact that this number comes from a company press release, it doesn’t say anything about the number of doctors using it to diagnose, just that they use it.

If a physician uses Google to search for a dosage chart for some drug they rarely prescribe, you wouldn’t say they are using Google to diagnose the patient. You wouldn’t say that either if they used Google to search for the most recent studies on a topic.

[sambellll]

To me this is like a good software engineer using AI.

The fact that they use it doesn't make what the result is any worse or less trustworthy - arguably it makes it better.

It only becomes a problem if they offload all of the thinking to AI.

[sublinear]

Human expertise is also improving all the time and not limited to just connecting dots. When AI seems to surpass a particular human, it's just because the human lacks broader knowledge and fails to investigate further.

An expert already knows they don't know everything. That was never the point. Critical thinking cannot be delegated to AI any more than it can be delegated to a book. There is nothing new going on here.

[qnleigh]

Totally agree. I'm a scientist, and like most scientists I have some specialized skills that most of my colleages don't. AI has empowered them to learn and build things that they might have otherwise needed me for. But there have been quite a few cases where it led them very far down a wrong path. This has started happening way more often in the last few months.*

We've known since the beginning that AIs confidently say incorrect things. But now that they can speak confidently about very complex topics, and mostly say correct things, we are letting our guard down and lots of subtle falsehoods are slipping through.

*In one case, I was able to put things back on track because the AI suggested my colleague talk to me; somehow it figured out we were co-workers.

[aspenmartin]

Right but hallucination rates have been consistently decreasing every model iteration. It's about error rates. As also a fellow scientist, I also will mess something up. Humans have an error rate. Once that error rate is low enough, it doesn't matter that it's > 0, it matters that it's low enough to be trustworthy and useful. Coding agents of 2024-25 had error rates too large; you couldn't meaningfully vibe code anything and needed a ton of oversight. It's still true but FAR less so, and this is after like a year of iteration.

[bitlad]

>very far down the wrong path.

Absolutely agree. Have seen this first hand

[sxg]

I see your argument, but it's not exactly news that an expert found a flaw in a popular tool. You could say the same about Wikipedia--experts have tons of issues with it, but Wikipedia still provides value to non-experts. The most likely alternative to Wikipedia for non-experts is simply not trying to learn anything new.

Similarly with LLMs, you can't just write them off entirely because they sometimes provide misleading or incorrect advice. The positive utility maximizing view is to learn when you need to call in an expert. I recently moved in to a new house and have used Claude extensively to figure out basic things (e.g., adjusting the garage door height, how to mount a TV). However, when the HVAC suddenly stopped working, I gave Claude a shot for an hour and tried some non-destructive fixes, but then realized I had to call in an HVAC expert.

[frereubu]

Slightly OT Nitpick: in regard to experts and Wikipedia, when doing a neuroscience-adjacent MSc, experts in the field actually directed me to Wikipedia as an excellent source for high-level neuroanatomy, including recent research, so I'm not sure your blanket description about experts and Wikipedia is correct.

[ohyes]

The free alternative to Wikipedia is the library, not “don’t learn anything new ever”.

I find Claude is surprisingly similar to a confident but incorrect coworker, with the benefit that Claude will reevaluate when I correct it.

[sxg]

I used the phrase "most likely alternative" intentionally. The library is where people should go to get answers in a world without Wikipedia, but the vast majority of people won't. So in practice, most non-experts either learn from Wikipedia or don't try to learn anything at all.

[ohyes]

Sure, if we’re going to go that broad. People are already leaning heavily towards learning nothing instead of using Wikipedia.

I guess to me it has to be comparable to be an alternative.

Like, I don’t consider doomscrolling x an alternative to reading Wikipedia but I might consider it an alternative to CNN, even though they’re all technically and very broadly activities that I could use to inform myself.

In that same way I don’t consider the multitude of ways I could use my free will necessarily alternatives to each other even though they technically are. It kinda sucks but going that broad feels to me like it breaks the concept of alternative and makes it kind of meaningless.

[sxg]

I get what you're saying, but I'm not deciding what should and shouldn't count as an alternative to X. I'm trying to answer the counterfactual: how do people behave in an alternative world without Wikipedia but otherwise identical to our world?

[bflesch]

Claude will do everything to retain you as a user, because that's one of their most important metrics.

[ohyes]

Excellent point my colleague has the exact opposite incentive.

[sbarre]

> Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

Yes, this is exactly so. AI is able to confidently sound plausible enough to convince laypersons or anyone who isn't very familiar with the subject matter, which is a big part of the mass-appeal "magic" of ChatGPT and other similar tools. It's like having a know-it-all friend (who also makes shit up to bridge their own knowledge gaps).

In many non-advanced non-specialized situations, AI is right enough to be at best useful or at worst not harmful (usually landing in the middle somewhere).

But speaking for myself, in areas where I consider myself quite proficient, I can very easily spot the subtle inconsistencies and naive conclusions that AI responses provide, and I have to guide/steer/correct it a lot to get good results when the subject matter is complex enough.

[scosman]

I dunno. I know a lot of software engineering experts. AI isn't always right, but neither are the people, and it's getting better and better.

Software is one domain where it excels because of structured training data and simulation environments, so I'm well aware it's better here than other areas.

Still there's somewhere balanced between saying every time it's "insufficient or incomplete or outright misleading" and "just trust AI". AI's a useful source of information/reasoning/research, but know you need to validate it's answers for important decisions.

[meowface]

I may be missing something, but I think it's unclear that the parent poster here is necessarily actually contradicting anything the AI said. It may depend on the exact information the OP wrote to Claude and GPT. The full transcripts would be needed. (Though there is definitely a separate point that a doctor would generally better know all the right questions to ask, while current LLMs may be making certain assumptions.)

The LLM may have, from its "perspective", implicitly thought the OP was telling it that he had strong reason to believe there was no calcification and was not considering the bigger picture of possibly receiving an incomplete/poor assessment from the medical staff. In fact, the issue here may be the LLM overly trusting doctors vs. trusting its own expertise.

[xivzgrev]

Well that's part of the problem. AI is not accountable - if you take its advice and hurt yourself, who is responsible?

A real doctor is accountable.

They might both "know" a lot of things but implicitly the party who is accountable is going to be more trustworthy.

And I don't see that going away until AI companies must be licensed for application x and can lose their license / be sued if engaging in malpractice.

[nlawalker]

> no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information

"Be wowed by the convenience and speed", or merely "take advantage of the mere availability"? What most people find to be damning about expert advice is that they simply can't get it anywhere, at any cost that they can afford.

[whatever1]

So if you want to do a surgery but you don’t see any surgeons around you ask a grocery butcher to have his way?

[sxg]

In certain circumstances, the answer is yes. If an airplane's pilots are incapacitated, do you simply give up and crash the plane because there are no other pilots on board? Or would you rather have someone on the ground try to coach a passenger into at least attempting to land the plane?

[frereubu]

That's an extreme edge case, which I don't think is in the context of the concerns in this thread.

[sxg]

The specific case doesn't matter--it's meant to make you think about the general question throughout this thread: when an expert isn't available, should non-experts use AI (or other tools) to help themselves? Sometimes the answer is yes because the potential benefits outweigh the potential harms (if any harms exist). But sometimes the answer is no because misleading/incorrect advice can cause a net harm.

[frereubu]

But if the cases where AI use is a net positive are one in a million in medical situations? The argument is surely about the ratio, which many people here are arguing (from anecdote, would be interested to see a real study) is not in its favour, and the potential downsides - from both false positives and negatives - can be huge.

[ChrisMarshallNY]

As long as that passenger didn’t have the fish.

[close04]

A passenger crashing the plane while trying to avoid a certain crash doesn’t make things any worse. An incompetent doctor trying to save you from certain death can make things so much worse. It’s all about weighing the best/worst outcome compared to where you are now.

[microgpt]

I hate to break it to you but death is certain for everyone.

Properly emotionally processing this fact and your complete inability to do anything about it is called an "existential crisis" and if you haven't had one or several yet, you will.

[close04]

I’m not sure what the “revelation” is? How is this related to what I said?

Putting that aside, your philosophy sounds shallow. Death is certain, but how long you have to live and the quality of that life are not predefined. An incompetent passenger-pilot trying to save you from a crash will at worst make no difference. But an incompetent doctor can teach you that death isn’t necessarily the worst outcome.

[jancsika]

You can choose a) a calm, level-headed passenger who knows they aren't a pilot, or b) a calm, level-headed passenger who almost has their pilots license but has a medical condition that prevents them from admitting when they lack certain knowledge.

Who do you choose to be coached by an expert on the ground?

[rvnx]

No thank you, I will ask Claude and then ask ChatGPT to challenge me, and do a couple of rounds like that.

The first: Has no clue about anything and therefore no useful knowledge and cannot challenge me

The second one: Is proven to willfully give wrong information and will make me do mistakes for sure.

The LLMs will do their best, even if imperfect, since they summarizes what appeared in books.

I prefer to be grounded on what Airbus / Boeing manuals, or on what pilots training book said, than two far more unreliable sources.

[perching_aix]

No, people don't even go to a butcher, they do it themselves if they can. See the countless stories about farmers and their inventiveness. Example: https://www.youtube.com/watch?v=KKaJhQBusH8

[EA-3167]

People, especially in medical crises, are desperate for answers that they often can't get because their clinicians don't know. The illusion of an all-knowing guru who sounds like their doctor and tells them ANYTHING is extremely alluring. If you're waiting to hear back from a doctor about test results (which these days probably showed up on your online account the moment they were completed) can be agonizing.

Ok for pain in your shoulder it might not, but how about a woman with a lump in her breast waiting for the mammogram interpretation? How about someone trying to understand disturbing lab results? People are also often pushed these days to move through visits with doctors at a breakneck speed, but the AI will "hear you out" all day.

Part of this is a problem with the AI, part of it a problem with our healthcare systems, and part of it is simply human nature. If you think that OpenAI, Anthropic, Google and the rest weren't aware of this going in you must have very little faith in the intelligence of their members. It's not hard to imagine the future of LLM's should involve a hell of a lot of liability on the companies running it, but for now it's the Wild West.

[bilsbie]

> but how about

Whatever scenario you come up with my answer is the same.

As an adult I’d like to be able to choose what tools I use to learn about my condition regardless of how well it works or even if it’s likely to mislead me.

There’s risk in every aspect of life and we can’t baby proof everything.

[baconmania]

>choose what tools I use to learn about my condition regardless of how well it works or even if it’s likely to mislead me.

Even if it "works" so poorly that you're not actually learning about your condition?

[EA-3167]

If it's helping you learn about your condition then sure I agree. The issue here is that's not really the case, it's giving you the illusion that you're learning about your condition while feeding you hallucinations and half-truths at best. A recent look at medical advice from these things showed they're no better than a coin flip.

So if you MUST have answers that are at most random guesses, I'd suggest saving a few bucks and asking a coin before flipping it.

[perching_aix]

The companies are 100% aware, yes, and so they did make quite a few changes over the years.

Current trend is that the models will try to explicitly steer you towards "asking better questions from your medical provider", rather than providing diagnoses. They do also evaluate whether something can actually be established rather than just listen and nod along. And so the "you must have very little faith in the intelligence of their members" goes right back against these failure mode ideas.

Now of course, given a sufficiently desperate person, they can probably torture anything they want to hear out of these models. But so can they out of actual people, so that's kind of a high bar. When you get to the point where people are willfully misreading a given piece of text, bets tend to be rather off.

[Aurornis]

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading. It is only when you do not know what the AI is being asked to do is it likely you will find the output helpful.

I always recommend people try asking LLMs a lot of questions on something they know first. Programmers should start by asking LLMs to work on a codebase they’re familiar with first.

You’re overstating the problem, though. Even for an expert the LLM will get a lot of things right and can be helpful under a watchful eye.

The real problem is knowing how to identify when it’s on the right track and when you need to correct it, because both cases are presented with the same tone and confidence.

An expert can better identify when the LLM output doesn’t sound plausible. Someone unfamiliar with the topic will think everything it says looks correct.

[highfrequency]

Seems natural enough. There will always be complexity and nuance that is missed by an AI model or person - the world is just super detailed. The more expertise you have the more you will be aware of that nuance. That doesn't mean the model or person is not useful as a starting point.

[serf]

>I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

media is awash at the moment with experts chiming in to support AI, saying their fields are being revolutionized, etc.

it seems unsurprising to me that the laymen opinion would follow the loudest media trumpets.

[rapatel0]

You shouldn’t expect frontier models to work on medical imaging. There is much more that goes into building a medical imaging product. First and foremost is data. Medical imaging datasets are not prevalent one the public internet at the scale necessary to have good performance on medical imaging tasks especially MRI. Also the labels are super noisy.

This is completely different than asking for general medical reasoning which is more derived from papers, public standards and textbooks.

Text exists at the right scale but images don’t.

[gofreddygo]

This is true in broader contexts too. Bunch of experts can't agree on something fundamental which is hard to prove/ disprove, and they have strong opinions on the topic.

AI is much worse.

[kryogen1c]

On the flip side of this problem, novel best practices lag the medical standard of care, other human failures like corruption and competing priorities notwithstanding.

For example, we had to advocate for certain practices during the birth of our first child that became routine during our second several years later.

So, neither side is guaranteed correct, doctor or citizen researcher (which did not include LLMs in my case, for the record). The truest answer is also the most useless one, applicable to all fields: it depends.

The real question is: if you embrace being a layman, whom do you trust more: LLMs/the internet or experts, like doctors? I think the answer is pretty clearly experts.

[mattgreenrocks]

You're not. This site was also bullish on using LLMs as therapists, which defeats the very point of them, and reflects a lack of knowledge on what exactly therapists do for people.

More on topic: if the article's author arrived at a definitively negative result would this have shown up on HN?

[je42]

The question is how far is AI off compared to the professional that we have access to. World best experts are not accessible to most of us. :(

[jstummbillig]

No, not anytime someone is an actual expert at anything, AI output appears insufficient. That is why experts in various fields use AI.

Then to say "Aha, but all of that is AI psychosis" makes obviously no sense: Why would we trust experts when they offer critique but not when they say "this is helpful"?

Overall: People are not insane. AI makes mistakes and, often, fails completely. AI also helps them do things better, quicker, increasingly so. The jaggedness of AI is confusing and real.

[torben-friis]

How many times have you seen an expert go "yeah these results are good consistently enough for a non expert to trust them without expert assistance"?

There is a huge difference between having a chance of a good result, which can be useful for experts able to filter out the bullshit, and consistent success. I would generate code as a helper, I would never allow a guy from marketing to merge unreviewed AI code.

[jstummbillig]

> How many times have you seen an expert go "yeah these results are good consistently enough for a non expert to trust them without expert assistance"?

But see now we are talking about something else entirely than the claim that I found dubious, which was: "Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading."

Consistently good enough !== anytime insufficient

[hectdev]

That's what I would like to call job security. When you know how to read what is wrong, you can easily catch the mistakes and correct it. AI gets you there faster by doing a lot of things right and you correct the mistakes.

[tpmoney]

I had a realization recently that the problem with "AI isn't consistently good enough" is that experience is probably not sufficiently distinguishable from the experience most non-experts have with computer systems all the time.

As an industry we've been promising people for decades that if they put all their data into our special softwares they can get all sorts of information back out that will make life easier for them, reveal new insights and otherwise improve their understanding. But the unspoken caveat has always been that you have to put the right data into the right places, in the right format, in the right way and then you have to ask the right questions, in the right syntax, with the right tools. And if you get any one of those parts wrong, you're not going to get the right answers (or possibly even any answer at all). How many people have had their excel worksheet that they (or someone else they asked/employed) built for some task that has been working fine for the last year suddenly stop working or start throwing out nonsense numbers because some input changed? Or how many people have experienced their system seemingly throw out meaningless garbage because daylight savings changed right at the moment the report was being run? Or spent months operating on wrong data because the person who wrote the query misplaced a parenthesis and the query was searching for "(foo AND bar) OR baz" and not "foo AND (bar OR baz)". For most people, the computer and the programs they use to do their jobs are magical black boxes that most of the time produce mostly the right answers and sometimes get things very very wrong with no indication of what has changed. Which is effectively the same experience they will have with an AI, but now instead of needing to figure out some arcane excel pivot table and VBA script, they can just dump some raw data and a "natural language" question into the AI.

And that's not counting the fact that their experience with looking information up online is about the same as well. How many absolutely confident wrong takes have you encountered online for things you're an expert in? How many of those wrong takes have come straight from supposedly trustworthy sources like news companies or even other people in the field?

For most people, using a computer has always come with the asterisk that you should always be aware that the source you're reading could be very wrong, that the output is only correct assuming all the inputs and all the parts processing that input are also correct and that everything you do should be accompanied by vetting by experts, whether those experts were software developers or domain experts. For most people the only thing that's changed with AI is that it's a one stop shop for their "probably directionally right, almost certainly wrong in the details" access to the digital oracles.

[lazide]

I’ve never seen an expert use AI in their field beyond the initial ‘oh interesting’ stage.

[jrockway]

I came here to post this as my experience. AI is magical when I apply it to something I know nothing about. It far exceeds my expectations every single time. I know nothing, but here is a report with animated graphics explaining exactly what I asked it to explain!

In fields where I'm an expert... it makes a lot of silly mistakes that are annoying and I feel like they would just cascade if I didn't correct them early. (I still think it's a net win, but... I watch it and it watches me, and we both do better work. I'd even apply the "magical" adjective when it does stuff I hate but know how to do, like edit Helm charts. What would normally be 20 minutes of me griping about YAML indentation is just a correct diff in seconds. I'll take it!)

So with that in mind, I tend to distrust output that I can't verify. If a doctor was recommending surgery and I thought the plan was too aggressive, I'd get a second opinion. I don't expect Claude Code to have much medical diagnostic ability, as that is really not what the model is trained for, and I know how it performs on work that it's trained and fine-tuned for. That is not to say the output is wrong and that it can't have diagnostic value, just that I personally wouldn't feel safe trusting it. Wrap up the same model with fine-tuning in the domain and a harness that reminds Claude to do a lot of sanity checks, perhaps with a human in the loop to guide it back onto the rails when it gets hyperfixated on something that doesn't matter? That could very much be a useful AI product.

[tomaskafka]

Yes. The PM’s “with AI I know enough to be dangerous, haha” means “I’m actually dangerous and I don’t realize”

[stymaar]

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading

AI assistant are industrializing the Gell-Mann amnesia effect.

[pwg]

> Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading.

The term for when the press "gets it wrong" is Gell-Mann Amnesia (https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect).

In that case, when you have personal knowledge of the facts, or know the specific domain area, you can see where the reporter mixed things up.

AI is no different, it's just a bunch of matrix math substituting for "the reporter" regurgitating what it was previously told. So the Gell-Mann Amnesia effect would apply just the same. If you have domain knowledge, you immediately see where the AI got it wrong. When you do not have domain knowledge, you have less chance of seeing where the AI was wrong.

[jefffoster]

AI is an expert in everything you are not.

[parineum]

> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading.

AI isn't even the first instance of this phenomenon, news articles are like this as well.

https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

[sevenzero]

>AI output appears insufficient or incomplete or outright misleading

It has been like this since the rise of "AI". The only people enthusiastic about it are usually the ones hoping to make a profit in one way or another.

[beering]

TFA doesn’t actually state where the bit about shockwave therapy came from and it wasn’t the main point of the article. The concern was about being given useless therapies. The homeopathic analgesic is concerning, at least to me.

I.e. nothing this radiologist said was related to the LLM’s advice.

[suttontom]

Your instinct is correct, and in a lot of cases it's true. However, I've heard from enough doctors by now (a cardiologist, psychiatrist, and epidemiologist/former physician) that they use medical LLMs and find them extremely helpful, mostly as a way to either bring up knowledge they'd forgotten about or as a way to learn something new and then verify it. I'm extremely skeptical about LLMs in general and the connection to Gell-Mann Amnesia is apt, but I wouldn't necessarily write them off completely like that. There are experts using the models that find them genuinely helpful in their field.

[GTP]

Probably this is the point, and it's a point that has been brought up a lot of times in the past, maybe less in recent times: you need to know the things you're applying an LLM to. In this way, you can keep the good outputs while having the expertise to discard the bad ones.

[Hikikomori]

It's like reading news articles. Seems reasonable until you read an article about something you know, then you see how wrong they can be.

[meindnoch]

We're past the point of Gell-Mann amnesia. This is full blown Gell-Mann psychosis.

[newsclues]

LLM is not necessarily an expert system. Once there are expert systems for law, healthcare, accounting, governance…

https://en.wikipedia.org/wiki/Expert_system

[microgpt]

Didn't they try that in the 80s and 90s but discover the real world is too variable for that to work?

[silisili]

This is natural and even logically expected. It's just Gell-Mann amnesia in action. The world has more people spouting on things than it has people knowledgeable in said things.

Apply that to the Internet at large, and realize where LLMs got their training. They're basically ConfidentlyIncorrect personified.

[stringfood]

what is happening is that the gap between what the experts and AI know is getting smaller each year. this year sure radiologists are mocking AI's ability to interpret MRI results, but they are a lot better at that this year than last. In five years perhaps radiologists will truly appreciate AI, but I am not holding my breath because radiologists are notoriously slow to adapt to changes in medical science compared to other specialists like anesthesiologists or surgeons

[grayhatter]

> This is itself alarming to me, but no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information.

Welcome to the club? This new awareness you've found over the true quality of LLM based GenAI output has been what "all the haters" have been mad about for-ever. That the output of LLMs are clearly defective, and merely have found a cute trick towards making humans think they're less defective than they are actually measured to be.

And the corresponding anger and frustration to push the risks of genai output out onto others, while also aggressively pushing it as a feature you should be using already. You're behind don't you know, and whatever other lie I have to tell to trick you into enough FOMO to pay me 200USD/mo so I can sell FOSS back to you.

An LLM can only output the mean next likely token, and then add a bunch of extra noise on top of that so it feels interesting and not repetitive. None of this is new, the problem is, 50% of humans are below the mean, but have no idea. So when an LLM tells them some lie: well, it sounds so helpful! It's impossible for someone who sounds this helpful to lie to me, liars never sound confident! It must be PERFECT! I'm gonna tell everyone how perfect it is. so the bottom 0-33% think LLMs are fantastic tools that make nearly 0 mistakes in comparison to the bottom 33%. 33-66%-ish aren't sure, some times it's great, but it will make that random mistake sometimes, but I can catch most (or all of them depending on ego). and the 66%+ are angry about how many people are getting tricked by something so obviously low quality, or are lucky enough to not have to care.

[orangecat]

An LLM can only output the mean next likely token, and then add a bunch of extra noise on top of that so it feels interesting and not repetitive.

So when an LLM was asked to analyze the unit distance conjecture, it just spat out a bunch of average-or-random tokens that coincidentally happened to correspond to a valid proof that had eluded humans for decades?