[wildermuthn]

“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.”

—-

Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI.

Fundamentally, he doesn’t define AGI correctly, and without a correct definition, his efforts to achieve it will be fruitless.

AGI is not about the degree of intelligence, but about a kind of intelligence. It is possible to have a dumb general intelligence (a dog) and a smart narrow intelligence (GPT).

When Ilya muses about GPT possibly being ephemerally conscious, he reveals a critically wrong assumption: that consciousness emerges from high intelligence and that high intelligence and general intelligence are the same thing. According to this false assumption, there is no difference of kind between general and narrow intelligence, but only a difference of degree between low and high. Moreover, consciousness is merely a mysterious artifact of little consequence beyond theoretical ethics.

AGI is a fundamentally different type of intelligence than anything that currently exists, unrelated and orthogonal to the degree of intelligence. AGI is fundamentally social, consisting of minds modeling minds — their own, and others. This modeling is called consciousness. Artificial phenomenological consciousness is the fundamental prerequisite for artificial (general) intelligence.

Ironically, alignment is only possible if empathy is built into our AGIs, and empathy (like intelligence) only resides in consciousness. I’ll be curious to see if the work Ilya is now doing on alignment leads him to that conclusion. We can’t possibly control something more intelligent than ourselves. But if the intelligence we create is fundamentally situated within a empathetic system (consciousness), then we at least stand a chance of being treated with compassion rather than contempt.

[abemiller]

I don't think it's fair for you to claim a leading AI expert has critically wrong assumptions when "musing" about possibilities that relate to one of the most epistemologically difficult topics to investigate (consciousness).

You're rejecting Ilya's humble musings as having critically wrong assumptions, and then turning around to definitively explain how consciousness arises, and illuminating the relationship between consciousness, empathy, and intelligence, on a random hacker news thread. Frankly, you're making some huge claims about philosophy of mind that don't obviously track for me, and you provide no citations or arguments to support. I hesitate to accuse you of "hallucinating facts", but when you're issuing a takedown of one of the top AI experts I'd expect to see some more supporting argument.

Your definition of AGI is also a bit strange as it requires that it be fundamentally different from existing natural intelligences, if I understand correctly. That seems unnecessarily stringent to me, since if a program had the same kind and level of intelligence as me, I'd be inclined to say it is AGI.

I'm just not sure where all these confidently stated, very specific claims are coming from.

[wildermuthn]

That’s a fair critique, and I appreciate the engagement. Thanks!

My thinking is based on the Attention-Schema theory of consciousness (AST), by Michael Graziano. His book “Consciousness and the Social Brain” is, I believe, the right roadmap for AGI. AST is basically a variant of the Global Workspace theory of consciousness, distinguished by its deterministic account of the mechanics and utility of consciousness.

“The Consciousness Prior” by Bengio also informs my thinking.

I’m not certain that I can point to anyone that has been as explicit as I have that phenomenological consciousness is a prerequisite for intelligence, but all the cookie crumbs are there for anyone interested in following the trail.

One correction to what you wrote — I’m explicitly saying that AGI will be fundamentally the same as existing biological intelligence, in that intelligence resides only in consciousness, and consciousness remains consciousness regardless of being biological or artificial. My point was that no currently existing DL models are generally intelligent.