[vharuck]

Right, the dialogue between the user and the LLM closely resembles documents used in training the LLM. People argue with, lie to, and misunderstand others on the internet. Here's a totally plausible hypothetical forum discussion:

Person A: I believe X.

Person B: Do you have a source for that?

A: Yes, it was shown by blah blah in the paper yada yada.

B: I don't think that study exists. Share a link?

A: [posts a URL]

B: That's not a real paper. The URL doesn't even work!

A: Works on my machine.

---

I've seen those kind of chats so many times online. Know what I haven't seen very often? When person A says "You're right, I made up that article. Let me look again for a real one, and I might change my opinion depending on what it says."