[Legend2440]

I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category.

https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...

  "kwargs": {
     "max_tokens": 100000,
     "stream": true,
     "reasoning_effort": "high",
     "rate_limit": {
        "rate": 2,
        "period": 60
     }
  }

The other paper ran Deepseek v3.2 without reasoning as a baseline and got 15.5%, which is much more in line with other base LLMs like GPT-5.2.

[Zababa]

Interesting, good find! Yeah I may be wrong and this may be an error in the leaderboard. Weirdly it shows no reasoning cost and no reasoning tokens used, but for example here https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... the answer is super short but it says "4945" completion tokens.