[asdfologist]

Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4).

It's more likely that the training data was contaminated with the benchmark data.

[mupuff1234]

You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check?

They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .

[goodmythical]

Is it really all that different from how the models "know" anything else?

The best way to know that christmas trees are typically purchased in december isn't to access historical financial tables and run the entire analysis yourself but to just "know it" from the articles you read attesting to the fact.

If you've found data that is the answer to the question, be it a random user question or a benchmark, the softwares' goal is to produce the correct solution and it is cheaper to retrieve from storage than to compute.

It's the same with coding. The agent usually isn't really thinking about the problem from first principles, it's just giving you the answers it's already found in instances where someone else asked the same question.

It seems to me that if you're really testing for reasoning capability, just as if you were an instructor administering a test, you'll need to change the test from run to run in order to make sure the agent/student isn't just copying old tests.

[johnfn]

How do you propose they check for something like this? They can't exactly ctrl-f the model weights for "Arc-AGI".

[vickychijwani]

Anthropic expends tons of compute and effort on understanding internal model states [1]; this kind of thing is right up their alley.

[1]: Recent example: https://www.anthropic.com/research/global-workspace

[phoghed]

> they can’t possibly know or find out what was in the training data

doesn’t appear to be a very strong argument