[SrslyJosh]

> I rarely trust code it generates (at least for the language I mainly work in) as i’ve seen it make some serious mistakes (eg: using keywords in the language that don’t exist)

It's only a mistake from your perspective. The model just generates text based the probabilities it learned during training. In that respect, there is no such thing as "incorrect" output because the model doesn't operate at that level of abstraction.

[tombert]

Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task.

I asked it "what is the PS3 game where the full version of To Kill a Mockingbird is in there?" and it responded back with "The Sabateour", when the correct answer would have been "The Darkness". That is incorrect by most definitions of the word, whether or not it's a consequence of the training model doesn't really change that.

I suppose we could get into details about epistemology and ontology about the nature of what an answer "is", but I think it's fair to say that "incorrect" is when it gives you something that doesn't accomplish the task you asked it to do, or rather when it tries to accomplish the task but what it gives you don't work.

[antonvs]

> Wait, no, it's "incorrect" in the sense that you asked it to do something, and the thing it gives you doesn't accomplish the task.

You believe you "asked it do something," but that's just you anthropomorphizing the model and your interaction with it. Of course the AI companies encourage that perspective, but it's a factually dubious one at best.

Judging whether a model's output is "correct" involves you imposing an external context on both the prompt and response that the model typically doesn't have access to. It also typically has no ability to test its responses.

This is part of why good prompt engineering can be so important - because what you get out is a function of what you put in, and pretending that the model is a question-answering oracle only takes you so far.

Of course what the AI companies are trying to do is train and prompt the models in such a way that their output is considered "correct" from a user's perspective more often than not. In an interaction with an AI company's salespeople, you might argue about "correctness". But that's not going to help understand what's actually going on.

[tombert]

It's actually not "anthropomorphizing the model".

I passed it input in the serialized form known as "English text". I expected a response also in serialized English that I can then decode in my brain to something that comports with reality. If I requested from a web server some JSON giving me my bank balance, and the balance it gave me is not accurately reflecting reality, it's not anthropomorphizing anything to say that it's incorrect, any more than pinging Nginx is.

And to be clear, we can wax philosophical all you want about "correctness", but that's really sidestepping the point: I don't care why it's giving me wrong information.

In my bank example, does it really matter, for the end user, if it's because of some integer overflow error or if it's a null pointer there's just a special `if` statement saying that antonvs account should always print out a different number for your balance.

I think nearly everyone would say that that's incorrect, and it actually wouldn't be clever or insightful for someone to say "no that's just a result of how the computer was programmed! You're imposing a human understanding of correctness on your bank balance!"

[antonvs]

> I expected a response also in ...

Exactly, you expected it, but that doesn't change what's actually happening. The model doesn't know what you expect. It can't read your mind. The best it can do is infer some things, such as that English input should produce English output - and the models are indeed pretty good at that!

> to something that comports with reality.

This is a rather unrealistic expectation in general, when you examine it. You raised a good example with which to do that, though:

> it actually wouldn't be clever or insightful for someone to say "no that's just a result of how the computer was programmed! You're imposing a human understanding of correctness on your bank balance!"

You're right, it wouldn't, because that's a very different situation which helps illustrate the point. The code for the bank app has been written to match your notion of correctness. That's only possible because it has a narrowly defined, specific purpose. It has all the necessary information needed to produce a correct response. The acceptance criteria are clear, including validation and integrity checks on the response. As a result, your expectations should be satisfied, and if they aren't, it makes sense to say that the bank app is not correct.

None of that applies to the AI models we're discussing. An LLM or image model doesn't have a narrowly defined, specific purpose. It can't possibly have access to all the information it needs to "answer" any possible "question" "correctly". It can't possibly have access to acceptance criteria specific to a question unless they're provided explicitly and in detail as part of a prompt - again, underscoring the importance of prompt engineering. And its ability to validate responses - check whether they "comport with reality" - is very limited, at least currently.

An example that's closer to the situation with an AI model would be a tool like a hammer. If you hold a hammer by its head and try to hammer in a nail with its handle, is the hammer "incorrect" when it fails at the task you have "asked" it to do?

> I don't care why it's giving me wrong information.

Just as with the hammer, if you want to be able to use these tools effectively, you should care why.

[tombert]

We’re talking about different things.

Let’s look at regular intelligence. Stuff in human brains.

If I ask a human a question like “how many sides does a square have?” and that human says “three”, that’s an objectively wrong answer, and I would say that that person is “wrong”.

The human brain also does not have a narrowly defined specialized purpose and can do a lot of things and yet we are perfectly ok still saying “true” or “false” to the truth values being asserted.

A truth proposition can have a “correct” or “incorrect” value. If a human gives me the wrong bank balance because they misremembered it, it’s still wrong regardless of the actual intelligence associated with it.

So again, I don’t think this kind of pedantry is actually useful because you’re not actually saying anything; you’re saying “these models aren’t trained on being correct, they’re on word patterns”, which was never disputed, but that is orthogonal to the truth value.

[rsynnott]

You're anthropomorphising too much. The machine did the correct thing; it's just that it is a prediction machine, not a magic question answering machine, so its 'correct' may not be what you wanted.

> I suppose we could get into details about epistemology and ontology about the nature of what an answer "is"

The machine has no concept of an 'answer'; when people call these things autocomplete on steroids, they're not really being that inaccurate.

[tombert]

It really isn’t anthropomorphizing at all. If I treat it like a black box it really is quite simple. I gave it an input in the form of English text, there is an objectively correct and incorrect form of English text that responds to the input, it gave me the incorrect one.

Literally no one here disputes that the it’s a glorified autocomplete, but that is completely irrelevant to if it correctly answered a question.

I find this kind of pedantry extremely annoying because it sounds insightful without actually saying anything. Like, no shit, it’s just doing what its algorithm dictates, no one, and I mean no one disputed that. The question of correctness and incorrectness falls into “how accurately did it respond to my query?”

[nomel]

This is like saying the arguments put forth by a schizophrenic lawyer are rational and correct.

If the context is that it's a tool, correct is defined as reality within the context of the use of that tool. If it's to find facts, it can be incorrect, since the context of a fact is reality. If it's writing a story, then "correct" would be based on continuity, etc.

If you're using it as a tool to generate words related to previous ones, then sure, it's always correct, but that's not probably not a useful tool for most people. But, being a next word predictor doesn't mean it can't also be a useful tool in real world contexts. There are, literally, billions of dollars being spent on pushing them to be more "correct" in more contexts, so it's a useful concept being considered, even though they're "just" next word predictors.

[LordKeren]

While yes, this is the technical reason — it’s important to not overlook how non-technical people see LLMs. And not only that, how they are being marketed.

I’m struggling to think of any comparable technology where the regular median users understanding is both fundamentally wrong— and is being purposefully misinformed.

[tikhonj]

That's like saying "there is no such thing as a bug, it's just code working the way it was written"—true in some sense, but not useful.

[williamcotton]

“Correctness” is a property of a proposition determined by an observer. Sometimes what is output by an LLM is correct, sometimes not. That an LLM is aware of the output or not means literally nothing.

[VS1999]

This habit of latching onto one word specifically to ignore what everyone knows is obnoxious, pedantic, and most of the time not even technically correct. It's just stupid quibbling over how words in English can be used to mean different things. And just so you know, the model doesn't "learn" anything, you're just adjusting weights until you get a desired result.

[elicksaur]

“What everyone knows” - https://www.lesswrong.com/posts/BNfL58ijGawgpkh9b/everybody-...

More broadly, the meaning and usage of specific words are important for these products because they shape how people perceive their utility.

If a thing isn’t “correct” because it has no sense of understanding, and therefore is only “correct” due to projection by the user, then that’s a super important distinction.