[ilaksh]

It's a really fascinating topic, but I wonder if this article could benefit from any of the extensive prior work in some way. There is actually quite a lot of work on AGI and cognitive architecture out there. For a more recent and popular take centered around LLMs, see David Shapiro.

Before that you can look into the AGI conference people like Ben Goertzel, Pei Wang. And actually the whole history of decades of AI research before it became about narrow AI.

I'd also like to suggest that creating something that truly closely simulates a living intelligent digital person is incredibly dangerous, stupid, and totally unnecessary. The reason I say that is because we already have superhuman capabilities in some ways, and the hardware, software and models are being improved rapidly. We are on track to have AI that is dozens if not hundreds of times faster than humans at thinking and much more capable.

If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use.

Don't get me wrong, I love AI and my whole life is planned around agents and AI. But I no longer believe it is wise to try to go all the way and create a "real" living digital species. And I know it's not necessary -- we can create effective AI agents without actually emulating life. We certainly don't need full autonomy, self preservation, real suffering, reproductive instincts, etc. But that is the goal he seems to be down in this article. I suggest leaving some of that out very deliberately.

[doctor_eval]

I don’t work in the field at all but

> it will actually out-compete us for resource control. And will no longer be a tool we can use.

I’ve never been convinced that this is true, but I just realised that perhaps it’s the humans in charge of the AI who we should actually be afraid of.

[ilaksh]

What I am proposing is to imagine that after successful but unwise engineering and improvements in hardware, there would be millions of digital humans on the internet, which emulate humans in almost every way, but operate at say 5 or 10 times the speed of humans. To them, actual people seem to be moving, speaking, and thinking in extreme slow motion. And when they do speak or do something, it seems very poorly thought out.

We should anticipate something like that if we really replicate humans in a digital format. I am suggesting that we can continue to make AI more useful and somewhat more humanlike, but avoid certain characteristics that make the AI into truly lifelike digital animals with full autonomy, self-interest, etc.

[fmbb]

We already have artificial persons in the world competing with humans for resources, and have had for hundreds of years: corporations.

[sonink]

> If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use.

I believe it is almost certain that we will make something like this and that they will out-compete us. The bigger problem here is that too few people believe this to be a possibility. And when this becomes certainty becomes apparent to a larger set of people, it might be too late to tone this down.

AI isn't like the Atom Bomb (AB). AB didn't have agency. Once AB was built we still had time to think how to deploy it, or not. We had time to work across a global consensus to limit use of AB. But once AI manifests as AGI, it might be too late to shut it down.

[mylastattempt]

I very much agree with this line of thought. It seems for humans it is the default mode of operation to just think of what is possible within the foreseeable future, rather than thinking of a reality that includes the seemingly impossible (at the time of the thought).

In my opinion, this is easily noticeable when you try to discuss any system, be it political or economical, that spans multiple countries and interests. People will just revert to whatever is closest to them, rather than being able to foresee a larger cascading result from some random event.

Perhaps this is more of a rant than a comment, apologies, I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere. Just thinking out what could happen in the next 5, 10 and 99 years.

[sonink]

> I suppose it would be interesting to have an online space to discuss where things are headed on a logical level, without emotion and ideals and the ridiculous idea that humanity must persevere.

Absolutely. Happy to be part of it if you are able to set it up.

[hollerith]

>the ridiculous idea that humanity must persevere.

Could you expand on what you mean by this? Specifically, is it OK with you if progress in AI causes the death of all the original-type human people like you and I?

[mylastattempt]

That comment was meant in a more general or universal sense. Perhaps consider it in the context of 'saving the earth'. There is no earth to be saved. The universe exists, and that's it. Life in all it's forms will find some way to survive. Or not. Wether it reverts all the way back to the size of insects or bacteria before it has a chance to flourish again, well, who knows, but so be it.

Positioning the animal known as 'human' as some God-like entity that must survive at all costs, is extremely arrogant if you ask me. Obviously I wish for humanity to thrive and survive, as this is self preserverance and a bit of pride or ego. But the notion that we are special in some way just rubs me the wrong way and doesn't help think ahead on a large scale and timeline.

[tivert]

> I believe it is almost certain that we will make something like this and that they will out-compete us. The bigger problem here is that too few people believe this to be a possibility. And when this becomes certainty becomes apparent to a larger set of people, it might be too late to tone this down.

I think the bigger problem is that too many people are focused on short term things like personal wealth or glory.

The guy who make the breakthrough that enables the AGI that destroys humanity will probably win the Nobel Prize. That potential Nobel probably looms larger in his mind than any doubts that his achievement is actually a bad thing.

They guy who employs that guy or productionizes his idea will become a mega-billionaire. That potential wealth and power probably looms larger in his mind than any doubts, too.

[hollerith]

That is why the government should help the researcher and the tycoon do the right thing by shutting down the AI labs and banning research, teaching and publishing about frontier AI capabilities.

[visarga]

> Once AB was built we still had time to think how to deploy it, or not.

It's in human hands, we can hardly trust the enemy or even ourselves. We already came close to extinction a couple of times.

I presume when ASI will emerge one of its top priorities will be to stop the crazies with big weapons from killing us all.

[mensetmanusman]

It can’t outcompete us on the global level due to energy restraints.

It would require a civilization to consciously bond with its capability to do so (in such a way that it enhances the survival of the humans serving it). Not sure this would be competition in the normal sense.

[rerdavies]

The problem will not be the AIs; the problem will be who owns the AIs, and how will we control them?

[Jensson]

> If people succeed in making that truly lifelike and humanlike, it will actually out-compete us for resource control. And will no longer be a tool we can use.

Symbiotic species exists, AI as we make them today will evolve as a symbiote to humans because its the AI that is most useful to humans that gets selected for more resources.

[zurfer]

The certainty in this sentence is not appropriate. You refer to an evolutionary process that is highly random and inefficient. We don't have millions of tries to get this right.

[mensetmanusman]

I think the certainty is warranted assuming we are talking about intelligence and not some sort of paper clip generator that does something stupid.

An intelligent entity will want to survive, and will realize humans are necessary cells for its survival. A big dog robot with nukes might not care, but I wouldn’t call that AI in the same sense.

[griffzhowl]

> An intelligent entity will want to survive

I don't see how that follows at all. You could have an intelligent entity, in the sense that you can ask it any question and it'll give a highly competent response, that is nevertheless completely indifferent about survival. Biological organisms have been formed by selection for survival and reproduction, so they have they operate accordingly, but I don't see the justification for generalising this to intelligences that have been formed by a different process. You would need some further assumptions on top of their intelligence - e.g. that their own actions are the dominant factor in whether they survive and replicate, rather than human interests.

[clob]

Apoptosis is when single cells die on purpose for the development of an organism. I don't think it's unreasonable to at least hypothesize this extending to intelligent entities. Humans aren't xylem cells, but perhaps Zorklovians from planet Fleptar do this.

Or, you know, Alan Turing eating the apple. I think he was a pretty smart guy.