| >> 2. The AI does TDD using those outputs to create its own copy of the game.- >> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human. In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful. Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game? >> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time. That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model. It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all. Have the ARC people (Chollet and friends) claimed that this result is cheating? |