Hacker News new | ask | show | jobs
by wongarsu 42 days ago
Because nearly all benchmarks measure "accuracy" by giving you a point for a correct answer, and 0 points for everything else. If you have 100 questions you are 10% certain on, answering "I don't know" to all of those leads to 0 points, answering all of them as if you are confident leads to an expected value of 10 points. So that's what most AIs are trained to do

AA-Omniscience is the only AI benchmark I know of where randomly guessing gets you a lower average score than answering all questions with "I don't know"
2 comments
jampekka 42 days ago
AA-Omniscience Index gives +100 for correct, 0 for "I don't know" and -100 for incorrect.

For your scenario the confident confident strategy will give average of -90. Saying I dont't know to all will give 0.

A lot of models have negative AA-Omniscience Index.

They also do have AA-Omniscience Accuracy and AA-Omniscience Hallucination Rate that handle "I don't knows" differently.

https://artificialanalysis.ai/evaluations/omniscience

link
nutjob2 42 days ago
It should be 1 for correct, 0 for don't know and -1 for wrong.

They are much better incentives. In real life a wrong answer is much more damaging than a don't know.

link
jampekka 42 days ago
"AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. Scores range from -100 to 100, where 0 means as many correct as incorrect answers, and negative scores mean more incorrect than correct."

https://artificialanalysis.ai/evaluations/omniscience

link
ExtremisAndy 42 days ago
See, this, to me, seems obvious, but I’m sure it’s more challenging/complex than I can imagine (I am NOT an expert on AI in any way imaginable). But there has to be a solution. Just yesterday I was asking Gemini to tell me about a certain college professor, and it gave me a list of facts about them. And it was perfect. Then, out of curiosity, I followed up with “tell me more about him!” and it spit out several more bits of information about this person that were entirely hallucinated (e.g., gave them credit for writing papers they didn’t write, said they won awards that actually someone else won). I know this is all complex and certainly beyond my limited skill set, but goodness, we’ve got to get this figured out with so many people depending on and trusting these things nowadays. It’s quite scary.
link
embedding-shape 42 days ago
I bet most of these issues are essentially system prompt/harness issues.

If your example had "Validate any details before sharing them with the user, with multiple sources" as the system prompt, it was using a model that is strong at following system prompts precisely and had access to some basic tools, then it'd spend maybe minutes more, but the answer would have been way more accurate.

But no, Google want "the new search results" (LLM hallucinations) to be on top, so we end up with "sounds plausible" answers instead "Collection of evidence from reliable/semi-reliable" or similar, which sucks. We could have quality, but it's too expensive/slow, so we get slop instead, just to maximize for speed and convenience.

link
techpression 42 days ago
Errors multiply though, you might just get more plausible sounding errors than actual facts.

Like when agent 1 says X, agent 2 verifies it as Y and the original question ends up being some weird amalgamation of Z with additional ”this is really true” statements sprinkled on top.

I agree Google responses hurt more than help, but I’ve also gotten identical outcomes of 40min self-reasoning Opus threads (it’s less common obviously).

link
embedding-shape 42 days ago
> Like when agent 1 says X, agent 2 verifies it as Y and the original question ends up being some weird amalgamation of Z with additional ”this is really true” statements sprinkled on top.

Yeah, seems what grounds agents right now is quite literally human thoughts and text, so if you're doing something like that, you really need to pass the original user prompt through the entire way, for every "child" to keep in mind the final thing, otherwise it does seem to spiral out of control.

link
bee_rider 42 days ago
Maybe some extra buckets could be added like depending on whether the answer ought to be known. Or, quality of the justification. “I don’t know and here’s a good reason why” is much better than “idk.” Correctly identifying that something is fundamentally unknown/unknowable is probably better than a simply-correct answer, even, right?
link
ipaddr 42 days ago
It should be -1, -.1, 1 because I don't know is slightly negative.
link
hhhAndrew 42 days ago
Interesting, I was about to say -1, 0.9, 1.0, because I don't know is almost as useful as the correct answer!
link
maxbond 42 days ago
And also because it creates "one neat trick" where it can answer "I don't know" for many/most things and still get credit.
link
tasuki 42 days ago
> In real life a wrong answer is much more damaging than a don't know.

I don't know. Is it?

link