[hughprime]

To quote the full paragraph:

I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.

So what he's really saying is that the question "Can machines think?" is too ambiguous since the words "machine" and "think" aren't well-enough defined. We could use a Gallup poll to find out what people think the words mean, and then try to answer the question based on those definitions, but that would be absurd.

What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners. Eventually somebody will write a chatterbot which is capable of fooling most of the judges into thinking it's human for the duration of a short conversation (if you've read the transcripts they rarely go on for more than a dozen lines or so) but it will be a bunch of clever tricks rather than an AI, and a serious attempt to figure out whether it's intelligent (e.g. by telling it a story and then asking it basic reading-comprehension questions about the story) will quickly show that it isn't.

[joe_the_user]

What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners.

Really?

If all we had to begin with were paper airplanes, then a paper airplane competition would certainly promote transcontinental airliners. Before powered flight, studying paper airplanes was certainly one valid way to make progress. And the limits of today's AI in many ways make it seem pretty close to paper airplanes - so seeing if they can every fly across a room seems entirely valid.

SHEESH, it's hard to find even a metaphor in which this controversial prize doesn't advance our knowledge.

Consider, human behavior is very ad-hoc. AI as a field has neglected ad-hoc interaction, preferring more logically specified activities. If a computer could hold an effective five minute chat, it would be so far above what currently exists as to be breath-taking. If that's done with 'tricks', it's time to start understanding the 'tricks' rather going on and on with the systematic or whatever approaches we might have thought were the proper way to do this.

Oh, and the reading test sounds easier than the interaction test - MUCH easier.

[hughprime]

My suspicion that it wouldn't be _that_ hard to be convincing in the context of the competition comes from reading some of the chat transcripts -- see, e.g. http://loebner.net/Prizef/2008_Contest/loebner-prize-2008.ht... for last year's winners. The conversations with actual humans seem fairly dull and formulaic, the judges don't seem to attempt any particularly sophisticated interactions. Already the winners are managing to convince about 30% of the judges, so getting that above 50% doesn't seem too hard.

Incidentally, I just had a chat with last year's winner elbot (see elbot.com) and it's got some pretty clever special-case interactions built in. For instance:

me: What is 4 times 4?

elbot: The answer is: 17!

me: Haha, very clever.

elbot: I'm glad you agree, most people think the answer is 16.

[joe_the_user]

If you think that doing this isn't hard, why attempt to do it! So far everyone's failed, you would certain get some attention even if it was all just 'tricks'.

[hughprime]

Oh, I think we're using the word "hard" in two different ways. I'm sure it would be very hard for me (or anyone else) to do, I'm just saying that it would be many orders of magnitude easier than building a true AI. Furthermore I don't think that one would be much of a step on the way to the other, since I really suspect that building a true AI is best done via either:

a) Reverse-engineering a human brain at the neural level, or b) Some kind of evolution

whereas I'm reasonably sure that the Loebner Prize could, given enough money, brains and time, be won by a large bag of special-case tricks based on careful observation of how the judges tend to behave in practice.