[jonathankoren]

My personal thoughts about AGI is that we’ll never “achieve” it for pretty much the same reasons you, and philosophers for hundreds, if not thousands, of years have said. We can’t even be sure that anyone else is conscious and intelligent, or just a clever facsimile.

As AI (in the broadest sense) has developed, we always end up moving the goal posts. Sometimes this is because we genuinely don’t know what is difficult and what is easy due to several billion years of evolution. But some of this is because we know how the system works, and so it can’t be “intelligence”.

I think of it as like a magic trick. When you watch a someone do an illusion well, it’s amazing. They made the coin disappear. It’s real magic! But then you find out all they did was stick in their pocket, or used a piece of elastic, and then “magic” is gone.

Essentially this is partially what the Chinese Room is about. You think the Chinese speaker is real, but then you find out it’s just some schlub executing finite state machine.