[seizethecheese]

Kimi doesn't do well on my "ask a trivia question that other AIs get wrong" test.

The question it came up with, "which U.S. state is closest to Africa?" is a pretty standard trivia question without any reason to believe other AIs would get confused. https://pellmell.ai/s/dccdeca69f929f79bc89317035610049

Even GPT-OSS-120b gets this right: https://pellmell.ai/s/1a43dfc7a3baa214aa0fa1b95d2c536a

[anigbrowl]

Are you giving it your API for these other AIs to evaluate their responses? This 'test' seems perverse.

[seizethecheese]

I don't understand the question.

The other AIs don't see the question until they are asked to react.

[anigbrowl]

Sorry, I should have said 'API key'. What I mean is, why do you consider it a reasonable test for an AI to guess what others AIs don't know?

[tossandthrow]

These types of tests are kind of moot as agentic harnesses are taking over.

IMHO an Ai is the llm plus it's harness.

A good harness would allow the llm to investigate on a map.

Just like the llm can use a python script to figure out how many r's there are in strawberry.

These tests are simply not that predictable of performance of the llm.

[seizethecheese]

The test here is not how close the state is to Africa, the test is coming up with a question that is hard for other AIs to answer.

[bel8]

I don't see how that invalidates their point.

Besides, n=1 benchmark seems like more of a coin toss.