[Sharlin]

Problem is people seriously believe that whatever GPT tells them must be true, because… I don't even know. Just because it sounds self-confident and authoritative? Because computers are supposed to not make mistakes? Because talking computers in science fiction do not make mistakes like that? The fact that LLMs ended up having this particular failure mode, out of all possible failure modes, is incredibly unfortunate and detrimental to the society.

[pera]

Last year I had to deal with a contractor who sincerely believed that a very popular library had some issue because it was erroring when parsing a chatgpt generated json... I'm still shocked, this is seriously scary

[TeMPOraL]

"SELECT isn't broken" isn't a new advice, and it exists for a reason.

[Suzuran]

My boss says it's because they are backed by trillion dollar companies and the companies would face dire legal threats if they did not ensure the correctness of AI output.

[tzs]

Point out to your boss that trillion dollar companies have million dollar lawyers making sure their terms of service put all responsibility on the user, and if someone still tries to sue them they hire $2000/hour litigators from top law firms to deal with it.

[buggy6257]

Your boss sounds hilarious naive to how the world works.

[Suzuran]

In a lot of ways he is, despite witnessing a lot of how the sausage is made directly. Honestly, I think at at least half of it is wanting to convince himself that the world still functions in ways that make sense to him rather than admit that it's mostly grifters grifting all the way down.

[direwolf20]

The Gervais Principle framework calls this type of person a Clueless. They sit in middle management as a buffer between the Sociopaths who run the world, and the Losers who know the world sucks but would just like to get their paycheck and go home. I'm surprised to hear this actually play out — the Gervais Principle doesn't seem very empirical.

[nathanaldensr]

The high-trust Boomer brain cannot comprehend the actual low-trust society of grifters in which we live.

[pluralmonad]

I don't agree with this blanket statement. The internet is low trust for lots of reasons, but regular (read small, proximal/spatiotemporally constrained) communities still exist and are not grifters all the way down. Acknowledging that distant strangers are not trustworthy in the traditional sense seems reasonable, but is categorically different than addressing natural social groups (small and local).

[OGEnthusiast]

Yes, and most young Americans are locked out of those small, high-trust suburbs due to high housing prices. So instead they get to experience the magic of low-trust America first-hand, hence the disconnect between the young and the boomers.

[TeMPOraL]

This is a good heuristic, and it's how most things in life operate. It's the reason you can just buy food in stores without any worry that it might hurt you[0] - there's potential for million ${local currency} fines, lawsuits, customer loss and jail time serving as strong incentive for food manufacturers and vendors to not fuck this up. The same is the case with drugs, utilities, car safety and other important aspects of life.

So their boss may be naive, but not hilariously so - because that is, in fact, how the world works[1]! And as a boss, they probably have some understanding of it.

The thing they miss is that AI fundamentally[2] cannot provide this kind of "correct" output, and more importantly, that the "trillion dollar companies" not only don't guarantee that, they actually explicitly inform everyone everywhere, including in the UI, that the output may be incorrect.

So it's mostly failure to pay attention and realize they're dealing with an exception to the rule.

--

[0] - Actually hurt you, I'm ignoring all the fitness/healthy eating fads and "ultraprocessed food" bullshit.

[1] - On a related note, it's also something security people often don't get: real world security relies on being connected - via contracts and laws and institutions - to "men with guns". It's not perfect, but scales better.

[2] - Because LLMs are not databases, but - to a first-order approximation - little people on a chip!

[ryandrake]

> It's the reason you can just buy food in stores without any worry that it might hurt you[0] - there's potential for million ${local currency} fines, lawsuits, customer loss [...]

We are currently facing a political climate trying to tear many of these safeguards down. Some people really think "caveat emptor" is some kind of natural, efficient, ideal way of life.

[miki123211]

> [1]

Cybersecurity is also an exception here.

"men with guns" only work for cases where the criminal must be in the jurisdiction of the crime for the crime to have occurred.

If you rob a bank in London, you must be in London, and the British police can catch you. If you rob a bank somebody else, the British police doesn't care. If you hack a bank in London though, you may very well be in North Korea.

[TeMPOraL]

That's a fair point, and I suppose it is a major reason cybersecurity looks the way it does. The Internet as it is ignores the jurisdictional borders. But I still think cybersec is going overboard with controls, constraining use cases where international cybercrime is not a major factor in the threat model.

[saratogacx]

For this logic I like to point out that every AI service has text that says, essentially "AI can be wrong, double check your answers". If you had the same disclaimer on your food "This food's quality is not assured" would you feel comfortable buying it or would you take pause until you've built up trust with the seller and manufacturer.

There's so much CYA because there is an A that needs C'ing

[johnnyanmac]

Just doesn't understand the scale of money.

Maybe a million dollar company needs to be compliant. A billion dollar company can start to ward off any loopholes with lawsuits instead of compliance.

A trillion dollar company will simply change the law and fight governments over the law to begin with, rather than worrying about compliance.

[rsynnott]

And just how many rs does your boss think are in strawberry?

[breakingcups]

If only every LLM-shop out there would put disclaimers on their page that they hope absolve them of the responsibility of correctness, so that your boss could make up his own mind... Oh wait.

[tveita]

I think people's attitude would be better calibrated to reality if LLM providers were legally required to call their service "a random drunk guy on the subway"

E.g.

"A random drunk guy on the subway suggested that this wouldn't be a problem if we were running the latest SOL server version" "Huh, I guess that's worth testing"

[TheSpiceIsLife]

There’s a non-zero number of people who would get a chuckle out of a browser extension at replaces every occurrence of LLM or AI with a random drunk guy on the subway .

[direwolf20]

It could be the same extension that replaces every occurrence of the cloud with my butt.

[TheSpiceIsLife]

That’s the one I trying to think of, I could remember the ‘butt’ part.

[anon_anon12]

People's trust on LLM imo stems from the lack of awareness of AI hallucinating. Hallucination benchmarks are often hidden or talked about hastily in marketing videos.

[wpietri]

I think it's better to say that LLMs only hallucinate. All the text they produce is entirely unverified. Humans are the ones reading the text and constructing meaning.

[Cthulhu_]

I don't remember exactly who said it, but at one point I read a good take - people trust these chatbots because there's big companies and billions behind them, surely big companies test and verify their stuff thoroughly?

But (as someone else described), GPTs and other current-day LLMs are probabilistic. But 99% of what they produce seems feasible enough.

[nullc]

> But 99% of what they produce seems feasible enough.

This being a big part of the problem-- their false answers are more plausible and convincing then the truth. The output almost always seems feasible-- true or not is an entirely different matter.

Historically when most things fail they produce nonsense. If they do they are producing something related to the truth (but perhaps biased or mis-calibrated). LLM output can be both highly plausible and unrelated to reality.

[pjc50]

Billions of dollars of marketing have been spent to enable them to believe that, in order to justify the trillions of investment. Why would you invest a trillion dollars in a machine that occasionally randomly gave wrong answers?

[pousada]

I think in science fiction it’s one of the most common themes for the talking computer to be utterly horribly wrong, often resulting in complete annihilation of all life on earth.

Unless I have been reading very different science fiction I think it’s definitely not that.

I think it’s more the confidence and seeming plausibility of LLM answers

[oneeyedpigeon]

People are literally taking Black Mirror storylines and trying to manifest them. I think they did a `s/dys/u/` and don't know how to undo it...

[direwolf20]

They codysld start by trying to dysndo it.

I'm sorry. That was a terrible joke.

[Sharlin]

Sure, but this failure mode is not that. "AI will malfunction and doom us all" is pretty far from "AI will malfunction by sometimes confabulating stuff".

[rsynnott]

In terms of mass exposure, you're probably talking things like Cmdr Data from Star Trek, who was very much on the 'infallible' end of the fictional AI spectrum.

[TheSpiceIsLife]

Data was also famous for getting things embarrassingly wrong, particularly when interacting with his human colleagues.

[TeMPOraL]

The stories I read had computers being utterly horribly right, which resulted in attempts (sometimes successful) at annihilate humanity.

[johnnyanmac]

This is probably more of a GAI achievement, but we definitely need confidence levels when it comes to making queries with factual responses.

But yes, look at the US c.2025-6. As long as the leader sounds assertive, some people will eat the blatant lies that can be disproven even by the same AI tools they laud.