[Tadpole9181]

The point of Arc-AGI-3 is to measure model performance. We already know that models can one-shot and iterate on very rudimentary game implementations. And, naturally, once it effectively has a copy of the source code, it can use that to play the game better.

This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.

And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.

[vessenes]

I think this is just too simplistic a take; Arc-AGI-1 was wide open to all models, harnesses, etc, and had quite a lot of innovative structures implemented by hobbyists. At the time, this was seen as a good thing (it was), because we don't know the best system architecture for all sorts of problems right now -> innovation is good.

The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.

[skeledrew]

Sounds like a distinction between sport and work. How useful is pure model performance if it's known that there are conditions in which even greater performance can be achieved on real tasks? How useful is it to know how fast/far a person can run if they can ride a bicycle or drive a vehicle?