Do you have any suggestions for a better approach of testing artificial intelligence? I mean, in a way that allows comparing different approaches and being a reasonable metric of progress.
I don't. I'm guessing -and it's nothing but a guess- that for every problem that can be solved with intelligence there exists a solution that does not require intelligence. I'm guessing in other words that intelligence is the ability to come up with solutions to arbitrary problems. If that's true then there's no way to test for intelligence by looking at the performance of a system at any particular task, or any finite set of tasks, and so there's no way to create a "test for intelligence".
My guess is supported by the experience that, in AI research, every time someone came up with a plausible test for intelligence, an AI system eventually passed the test only to make it clear that the test was not really testing intelligence after all (edit: I don't just mean formal tests; e.g. see how chess used to "require intelligence" right up until Deep Blue vs Kasparov).
Some people see that as "moving the goalposts" and it's certainly frustrating but the point is that we don't know what intelligence is, exactly, so it's very hard to test for its existence or not, or to measure it.
My preference would be for everyone in AI research to either stop what they're doing and try to understand what the hell intelligence is in the first place, to create a theory of intelligence so that AI can be a scientific subject again, or to at least admit they're not interested in creating artificial intelligence. I, for example, am not, but all my background is in subjects that are traditionally labelled "AI" so I have to suck it up, I guess.
I watched the podcast you're referencing, on youtube, but I don't remember Chollet saying anything like what I say above.
Quite the contrary, Chollet seems convinced that a test for artificial intelligence, like an IQ test for AI, can be created and he has not only created one but also organised a Kaggle competition on it, and now is offering a $1 million prize to solve it. So how is anything he says or does compatible with what I say above, that it's likely there can't be a test for artificial intelligence?
You clearly didn't listen much since Chollet's point is exactly that there is no one task that can test for AI. The test he created is supposed to take that into account.
First, don't be a jerk. Second, what's your problem? That you think I said that "no single task can't be used to test for AI"? I initially said:
>> "If that's true then there's no way to test for intelligence by looking at the performance of a system at any particular task, or any finite set of tasks, and so there's no way to create a "test for intelligence"."
Stress on or any finite set of tasks.
So, no, I didn't refer to a single task, if that's what you mean. What the hell do you mean and what the hell is your problem? Why is everyone always such a dick in this kind of discussion?
Sorry. You, to me, came across as more interested in sharing your opinion than understanding what other people are saying. That's annoying. Maybe that's on me though.
Ok, you think no finite set of tasks can be used. Chollet is trying anyways. Maybe he is actually dynamically creating new tasks in the private set every time someone evaluates.
My main point was that I still think you're saying very similar things, quoting from the paper I mentioned:
> If a human plays chess at a high level, we can safely assume that this person is intelligent, because we implicitly know that they had to use their general intelligence to acquire this specific skill over their lifetime, which reflects their general ability to acquire many other possible skills in the same way. But the same assumption does not apply to a non human system that does not arrive at competence the way humans do. If intelligence lies in the process of acquiring skills then there is no task X such that skill at X demonstrates intelligence, unless X is a meta task involving skill acquisition across a broad range of tasks.
This to me sounds very similar to what you said:
> I'm guessing in other words that intelligence is the ability to come up with solutions to arbitrary problems.
My guess is supported by the experience that, in AI research, every time someone came up with a plausible test for intelligence, an AI system eventually passed the test only to make it clear that the test was not really testing intelligence after all (edit: I don't just mean formal tests; e.g. see how chess used to "require intelligence" right up until Deep Blue vs Kasparov).
Some people see that as "moving the goalposts" and it's certainly frustrating but the point is that we don't know what intelligence is, exactly, so it's very hard to test for its existence or not, or to measure it.
My preference would be for everyone in AI research to either stop what they're doing and try to understand what the hell intelligence is in the first place, to create a theory of intelligence so that AI can be a scientific subject again, or to at least admit they're not interested in creating artificial intelligence. I, for example, am not, but all my background is in subjects that are traditionally labelled "AI" so I have to suck it up, I guess.