[Y_Y]

I asked it if giraffes were kosher to eat and it told me:

> Giraffes are not kosher because they do not chew their cud, even though they have split hooves. Both requirements must be satisfied for an animal to be permissible.

HN will have removed the extraneous emojis.

This is at odds with my interpretation of giraffe anatomy and behaviour and of Talmudic law.

Luckily old sycophant GPT5.1 agrees with me:

> Yes. They have split hooves and chew cud, so they meet the anatomical criteria. Ritual slaughter is technically feasible though impractical.

[Flere-Imsaho]

Models should not have memorised whether animals are kosher to eat or not. This is information that should be retrieved from RAG or whatever.

If a model responded with "I don't know the answer to that", then that would be far more useful. Is anyone actually working on models that are trained to admit not knowing an answer to everything?

[spmurrayzzz]

There is an older paper on something related to this [1], where the model outputs reflection tokens that either trigger retrieval or critique steps. The idea is that the model recognizes that it needs to fetch some grounding subsequent to generating some factual content. Then it reviews what it previously generated with the retrieved grounding.

The problem with this approach is that it does not generalize well at all out of distribution. I'm not aware of any follow up to this, but I do think it's an interesting area of research nonetheless.

[1] https://arxiv.org/abs/2310.11511

[robrenaud]

Benchmarks need to change.

There is a 4 choice choice question. Your best guess is the answer is B, at about 35% chance of being right. If you are graded on fraction of questions answered correctedly, the optimization pressure is simply to answer B.

If you could get half credit for answering "I don't know", we'd have a lot more models saying that when they are not confident.

[anonym29]

>Models should not have memorised whether animals are kosher to eat or not.

Agreed. Humans do not perform rote memorization for all possibilities of rules-based classifications like "kosher or not kosher".

>This is information that should be retrieved from RAG or whatever.

Firm disagreement here. An intelligent model should either know (general model) or RAG-retrieve (non-general model) the criteria for evaluating whether an animal is kosher or not, and infer based on knowledge of the animal (either general model, or RAG-retrieval for a non-general model) whether or not the animal matches the criteria.

>If a model responded with "I don't know the answer to that", then that would be far more useful.

Again, firm disagreement here. "I don't know" is not a useful answer to a question that can be easily answered by cross-referencing easily-verifiable animal properties against the classification rules. At the very least, an intelligent model should explain which piece of information it is missing (properties of the animal in question OR the details of the classification rules), rather than returning a zero-value response.

To wit: if you were conducting an interview for a developer candidate, and you asked them whether Python supports functions, methods, both, or neither, would "I don't know" ever be an appropriate answer, even if the candidate genuinely didn't know off the top of their head? Of course not - you'd desire a candidate who didn't know to say something more along the lines of "I don't know, but here's what I would do to figure out the answer for you".

A plain and simple "I don't know" adds zero value to the conversation. While it doesn't necessarily add negative value to the conversation the way a confidently incorrect answer does, the goal for intelligent models should never be to produce zero value, it should be to produce nonzero positive value, even when it lacks required information.

[embedding-shape]

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

[latexr]

> How many times did you retry (so it's not just up to chance)

If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

[zamadatix]

If you retry until it gives the answer you want then it only serves to amplify your bias. If you retry and see how often it agrees with itself then it serves to show there is no confidence in an answer all around.

It's a bit of a crutch for LLMs lacking the ability to just say "I'm not sure" because doing so is against how they are rewarded in training.

[oivey]

You’re still likely to just amplify your own bias if you don’t do some basic experimental controls like having some preselected criteria on how many retries you’re going to do or how many agreeing trials are statistically significant.

[observationist]

https://en.wikipedia.org/wiki/Monte_Carlo_method

If it's out of distribution, you're more likely to get a chaotic distribution around the answer to a question, whereas if it's just not known well, you'll get a normal distribution, with a flatter slope the less well modeled a concept is.

There are all sorts of techniques and methods you can use to get a probabilistically valid assessment of outputs from LLMs, they're just expensive and/or tedious.

Repeated sampling gives you the basis to make a Bayesian model of the output, and you can even work out rigorous numbers specific to the model and your prompt framework by sampling things you know the model has in distribution and comparing the curves against your test case, giving you a measure of relative certainty.

[latexr]

Sounds like just not using an LLM would be considerably less effort and fewer wasted resources.

[dicknuckle]

It's a way to validate the LLM output in a test scenario.

[embedding-shape]

Well, seems in this case parent did know the answer, so I'm not sure what your point is.

I'm asking for the sake of reproducibility and to clarify if they used the text-by-chance generator more than once, to ensure they didn't just hit one out of ten bad cases since they only tested it once.

[latexr]

> so I'm not sure what your point is.

That your suggestion would not correspond to real use by real regular people. OP posted the message as noteworthy because they knew it was wrong. Anyone who didn’t and trusts LLMs blindly (which is not a small number) would’ve left it at that and gone about their day with wrong information.

[embedding-shape]

> That your suggestion would not correspond to real use by real regular people.

Which wasn't the point either, the point was just to ask "Did you run one prompt, or many times?" as that obviously impacts how seriously you can take whatever outcome you get.

[Y_Y]

Sorry I lost the chat, but it was default parameters on the 32B model. It cited some books saying that they had three stomachs and didn't ruminate, but after I pressed on these points it admitted that it left out the fourth stomach because it was small, and claimed that the rumination wasn't "true" in some sense.

[mistrial9]

due to reforms around the first centuries of the Common Era, trivia questions to certain tribal priests are no longer a litmus test for acceptable public goods in the marketplace.