[DtNZNkLN]

The lawyer said: “I did not comprehend that ChatGPT could fabricate cases."

I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter...

I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.

[IIAOPSW]

"hey chatGPT, can you make up some bullshit for me?"

"I'm sorry, as an online AI language model I am not able to generate output that is knowingly incorrect. Is there anything else you want help with today".

see, it isn't capable of lying. It told me so itself.

[jakelazaroff]

You say that as a joke but the lawyer literally asked ChatGPT whether the cases it provided were fake! There are screenshots! https://simonwillison.net/2023/May/27/lawyer-chatgpt/

Lawyer: Is varghese a real case

ChatGPT: Yes, Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019) is a real case.

Lawyer: What is your source

ChatGPT: I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion my earlier responses may have caused.

Lawyer: Are the other cases you provided fake

ChatGPT: No, the other cases I provided are real and can be found in reputable legal databases such as LexisNexis and Westlaw.

Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.

[jstarfish]

> Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.

Because that's work and takes effort. He gets paid the same to delegate the work to AI.

He did the absolute, bare minimum amount of verification needed to [hopefully] cover his ass. He just didn't expect the system to lie (sorry, "hallucinate") to him more than once.

> [...] the lawyers did not act quickly to correct the bogus legal citations when they were first alerted to the problem by Avianca’s lawyers and the court. Avianca pointed out the bogus case law in a March filing.

This is what fraud looks like. He's so checked out he even ignored the red flags being waved in his face. It stopped being a cute case of a student generating a common essay about Steinbeck when he started getting paid $200 an hour to cheat an injured client.

[unyttigfjelltol]

> It stopped being a cute case of a student generating a common essay about Steinbeck when he started getting paid $200 an hour to cheat an injured client.

It's more likely these lawyers are working on contingency and, because they did poor work, will receive nothing for it.

[jstarfish]

Good point!

[sokoloff]

> He gets paid the same to delegate the work to AI.

If he was being paid hourly, he would actually get paid more to go look up those cases in a database.

[jstarfish]

Well, yes, but you're assuming good faith in implying he's willing to spend his time on it. The point is to maximize hours billed while doing as little work as possible.

No contractor charges you for 2 minutes of work installing a $0.99 part; they pad it every way possible with service call fees, labor, etc. Attorneys just lie about it altogether since for logical work, you can't prove whether or not they actually did anything. It's all showmanship. Question them on it and it's all gaslighting about how you're not a lawyer and don't know what you're talking about.

Sibling comment points out possible contingency basis, so if true, he certainly wouldn't want to spend real time on a case that may not pay out. But if he can automate the process and collect winnings while doing no real work, it's a money printer.

[uoaei]

The new LLMs have been advertised in layman circles often as something like "a chatbot with access to all the information on the web" or something similarly ambiguous. So it is certainly easy to imagine why people think it literally goes out and checks its sources by re-accessing the webpage and summarizing it for the user. The responses you quoted seem to simulate/imply that kind of behavior through its natural language presentation.

[jakelazaroff]

Yeah, I've spoken to people who've had trouble internalizing that it can't search the web even after I've explicitly told them that. The "upon double-checking" message from ChatGPT is especially egregious — it's straight-up lying about how it arrived at that response. There really should be a layer on top of the chat UI to alert the user if it detects a response like that.

[lt_kernelpanic]

He assumed that ChatGPT was under oath, apparently.

[atmavatar]

"Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error."

[belter]

" Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error." - https://youtu.be/CD9YqdWwwdw

[blitzar]

"There is no danger that Titanic will sink. The boat is unsinkable and nothing but inconvenience will be suffered by the passengers." Phillip Franklin, White Star Line vice-president, 1912

[dunham]

For me it was perfectly willing to:

"produce fake technical language in the style of star trek"

[ilyt]

Inventing stuff out of thin air isn't technically lying

[taco_emoji]

It is literally just a goddamn language model. it is very good at making plausibly human-like sentences. It is not a general intelligence, it is not your friend, it is not a research assistant. It is not designed to deliver content which is correct, it is designed to deliver content which is similar to human language.

It might get things correct most of the time! But that is purely incidental.

[kevin_thibedeau]

It does subsume a corpus of factual information. I use it as a search tool for topics and relationships that traditional search engines can't handle. You just have to know that whatever it outputs isn't trustworthy and needs to be verified.

[jstarfish]

Part of the corpus is explicit bullshit though, and we don't know to what degree. It internalized conspiracy theory and WebMD alike. In a generative capacity, it only reliably produces fiction. Ever. Fictional stories often take place in realistic settings and reference real facts. They sound real. But they're still fictional compositions.

Using GPT as a reference to anything is the same as using a Michael Crichton novel as a biology reference. It looks right, but why would you waste your time asking questions of something you can't trust and have to double-check everything it says anyway? Nobody would keep an employee like that around, nor would you hang out with someone like that. It's friendly enough, but it's a pathological liar.

There's too much black magic going on inside the black box. We don't know how prompts get tampered with after submission, but it might be worth it to pepper "nonfiction" tokens in prompts to ensure it skews on the right side of things. It certainly responds to "fiction" when you're explicit about that.

[dekhn]

yes but it's still very much just a language model, not a knowledge model.

[travisjungroth]

It is literally just a goddamn electric motor. it is very good at converting chemical energy into mechanical energy. It is not a universal engine, it is not your horse, it is not your servant. It is not designed to create movement which is correct, it is designed to create movement which is similar to a piston engine.

It might move you forward most of the time! But that is purely incidental.

[verdagon]

I believe electric motors _are_ actually intended and advertised to create movement which is correct, and come with warranties and liability about reliability. (Or maybe I missed a joke in here somewhere?)

[seydor]

now imagine believing that this electric motor can self-drive itself to the supermarket

[travisjungroth]

ChatGPT isn’t just an LLM. It’s literally not. There’s a web server, interfaces, plugins, etc.

LLMs are this super powerful thing (like a motor) and people are getting to play around with it before it’s fully harnessed. There’s this strange phenomenon where because it’s not totally harnessed, people just rip on it. I don’t know if they think it makes them sound smart, but it sure doesn’t to me. It’s like seeing a motor on an engine stand and being like “But the crankshaft rotates. I want to go in a straight line! This isn’t a transportation solution and anyone who thinks so is just naive. And horsepower? Stop zoomorphizing it!”

Sure, there are lots of problems. But don’t mix up the limitations of this component of the system, the current limitations of the system overall, and the possible capabilities of the system. Someone builds a car that stalls when it drives through a river and people point out that an ICE engine needs air and how could you even think about driving through a river? Then someone else is off making a snorkel.

[vba616]

I think you've got hold of the wrong end of your analogy.

If you just hold it the right way you will be illuminated and we will all agree how relevant it is.

If someone announces to the public, they have a revolutionary new motor that is going to transform personal transportation and solve climate change...

...but it is presented with a chat interface - the motor itself is directed through typing text and it demonstrates its motion/power through text messages in response - rather being connected to a vehicle or generator, rather than a transmission, then it's BS beyond a reasonable doubt.

Even though there is no logical or technical reason preventing such a setup, it's just not how professionals do things. If the motor were so damn useful, then people who like to use motors would use it for the things that they do use motors for!

There is no logical or legal requirement in science to use LaTeX Computer Modern fonts...but as a layperson if I can't tell if someone is a crank or doing advanced physics, I'm going to be exponentially more skeptical if they are using Times or Palatino.

[travisjungroth]

Judging them for having a chat interface is the exact type of thing that I think is unintelligent. It would be like… judging an extremely popular paper for their choice of font. We’re past that.

[boredumb]

I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.

[Avicebron]

But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure

[taberiand]

I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards.

It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.

[Avicebron]

>I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards.

Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus

[seydor]

notOpenAI told them that this is so incomprehensibly smart that it will destroy all of us. Not because it's dumb and connects the wrong cables, but because it's super-smart. People can't surmise from that that the model just makes stuff up on the way

[JohnPrine]

OpenAI is clear about the fact that ChatGPT can hallucinate, where have they ever said otherwise?

[seydor]

One of the most widely circulated PR about GPT4 is that it passed the bar exam

https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-b...

and it's had the most vocal and sensational PR about how AI needs regulation now

https://edition.cnn.com/2023/06/09/tech/korea-altman-chatgpt...

People assume the smalltext does not apply to them.

openAI does not get a lot of flak from the media for the amount of BS that chatGPT can blurt out

Does anyone remember what happened to Galactica which did the same thing ? That too was clearly labeled as hallucinatory . But it was shut down because they did not BS the media enough about regulations and such.

I m afraid these LLMs are turning into too much of a political game to be useful for much longer.

On the other hand, if they become political, then people will be even more incentivized to build offline, local LLMs

[arbitrarian]

This is what I don't get. Not only have they not said otherwise, but they put it right up front in a pretty easy to understand brief message before you start using it. I guess lawyers just click agree without reading too.

[jprete]

People are accustomed to ignoring the fine print as legal CYA with no real-world relevance. This is also why the product warnings that "The State of California considers this to cause cancer" are a joke and not a useful message.

[Too]

It’s not even fine print, there is a big Limitations section almost at the top of the chatgpt landing page with this as first bullet:

”ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”

It does admittedly sound innocent, it should be written with a more scary tone on to how big extent it bullshits, not just “sometimes”. Still doesn’t excuse the incompetence of the lawyer in question.

[IIAOPSW]

I prefer to use the term "clopenAI"

[SilasX]

Or OpenAladeen

https://m.youtube.com/watch?v=NYJ2w82WifU

[kcplate]

> I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter...

AI isn’t going to destroy us in some Terminator/Matrixy scenario. We are gonna be the agents of our own demise by just blindly walking off a cliff created from our own laziness.

[CPLX]

Normal people don’t get that intuitively because it’s not intuitive at all.

For everyone who his looking down on this guy have you ever read a story on the Reddit front page and thought it was a real story accurately recounted by a real person and not a work of fiction? If so you’re equally naive.