[yaj54]

LLMs would be better nomenclature than AI in this context.

LLMs are not factual databases. They are not trained to retrieve or produce factual statements.

LLMs give you the most likely word after some prior words. They are incredibly accurate at estimating the probabilities of the next word.

It is a weird accident that you can use auto-regressive next word prediction to make a chat bot. It's even weirder that you can ask the chatbot questions and give it requests and it appears to produce coherent answers and responses.

LLMs are best thought of as language generators (or "writers") not as repositories of knowledge and facts.

LLM chatbots were a happy and fascinating (and for some, very helpful) accident. But they were not designed to be "factually correct" they were designed to predict words.

People don't care about (or are willing to accept) the "wrong answers" because there are enough use cases for "writing" that don't require factual accuracy. (see for instance, the entire genre of fiction writing)

I would argue that it is precisely LLMs ability to escape the strict accuracy requirements of the rest of CS and just write/hallucinate some fiction that is actually what makes this tech fascinating and uniquely novel.

[chrisjj]

> LLM chatbots ... were not designed to be "factually correct" they were designed to predict words.

For this question, what LLMs were designed for is I think less relevent than what they are advertised for, e.g.

"Get answers. Find inspiration. Be more productive. Free to use. Easy to try. Just ask and ChatGPT can help with writing, learning, brainstorming, and more." https://openai.com/chatgpt/

No mention of predicting words.

[mmusson]

The thing I find fascinating is that apparently there is a chunk of behavior that we might define as “intelligent” on some level that seems directly encoded in language itself.

[yaj54]

I completely agree. As language is the preferred encoding method for intelligent thought (at least in our species) it could very well be that a sufficiently accurate language model is also a generally intelligent model.

[tivert]

> LLMs are best thought of as language generators (or "writers") not as repositories of knowledge and facts.

And the utility of a "language generator" without reliable knowledge or facts is extremely limited. The technical term for that kind of language is bullshit.

> People don't care about (or are willing to accept) the "wrong answers" because there are enough use cases for "writing" that don't require factual accuracy. (see for instance, the entire genre of fiction writing)

Fiction, or at least good fiction, requires factual accuracy, just not the kind of factual accuracy you recalling stuff from an encyclopedia. For instance: factual accuracy about what it was like to live in the world in a certain time or place, so you can create a believable setting; or about human psychology, so you can create believable characters.

[yaj54]

I'd argue that what you're talking about in fiction is coherence (internal consistency) not factual accuracy (consistency with an externally verifiably ground truth).

I'd also argue that the economic value of coherent bullshit is ... quite high. Many people have made careers out of producing coherent bullshit (some even with incoherent bullshit :-).

Of course, in the long run, factual accuracy has more economic value than bullshit.

[tivert]

> I'd argue that what you're talking about in fiction is coherence (internal consistency) not factual accuracy (consistency with an externally verifiably ground truth).

No. I'm talking about "factual accuracy (consistency with an externally verifiably ground truth)." Mere internal consistency is not enough: a fictional world where everyone consistently stabs themselves in the eye when they see flashing lights is consistent, but lacks factual accuracy, and is therefore garbage fiction.

> I'd also argue that the economic value of coherent bullshit is ... quite high. Many people have made careers out of producing coherent bullshit (some even with incoherent bullshit :-).

I agree there's (greedily selfish) "economic value" to coherent bullshit, but there's negative social value to it. It's basically a kind of scam.

IMHO, some of the best applications for LLMs are for things like spam and scams, not the utopian BS they're promoted for (e.g. some LLM will diagnose your illness better, faster, and cheaper than a doctor).