[fchollet]

Right, one rather uninteresting line of approaches to ARC consists of trying to anticipate what might be in the test set, by generating millions of synthetic tasks. This can only work on relatively simple tasks, since the chance of task collision (between the test set and what you generate) is very low for any sophisticated task.

ARC 2 will improve on ARC 1 by making tasks less brute-forceable (both in the sense of making in harder to find the solution program by generating random programs built on a DSL, and in the sense of making it harder to guess the test tasks via brute force task generation). We'll keep the human facing difficulty roughly constant, which will be controlled via human testing.

[versteegen]

Hi! As someone who spent the last month pouring myself into the ARC challenge (which has been lots of fun, thanks so much for creating it), I'm happy to see it made harder, but please make it harder by requiring more reasoning, not by requiring more human-like visual perception! ARC is almost perfect as a benchmark for analogical reasoning, except for the need for lots of image processing as well. [Edit: however, I've realised that perception is representation, so requiring it is a good thing.]

Any plan for more validation data to match the new harder testset?

[Skylyz]

I had never thought about how close perception and reasoning are from a computational point of view, the parts of ARC that we call "reasoning" seem to just be operations that the human brain is not predisposed to solve easily.

A very interesting corollary is that the first AGIs might be way better thinkers than humans by default because of how they can seamlessly integrate new programs into their cognition in a perfect symbiosis with computers.

[versteegen]

Perception is the representation of raw inputs into a form useful for further processing, but it is not a feed-forward computation. You repeatedly re-represent what you see as you keep looking. Particularly something like an ARC puzzle where you have to find a representation that reveals the pattern. That's what my ARC solver is about (I did not finish it for the deadline).

> A very interesting corollary is that the first AGIs might be way better thinkers than humans by default

I agree at least this far. Human System 2 cognition has some very severe limitations (especially working memory, speed, and error rate) which an AGI probably would not have. Beyond fixing those limitations, I agree with François that we shouldn't assume there aren't diminishing intelligence returns to better mental architectures.