[trashtester]

Exactly. All that is needed for AGI to eventually be developed, is that humans do NOT have some magical or divine essence that set us apart from the material world.

Now the _timeline_ of AGI could be anything from the a few years to millennia, at least if evaluated 40 years ago. Now, though, it really doesnt seem very distant.

[akomtu]

That's going to be the key principle of the new religion invented by AI: that humans are just like machines, and AI is the supreme machine.

[trashtester]

Well, assuming superintelligence emerges AND we don't find any evidence of anything supernatural inside human brains, what does "machine" even mean at that point?

Anyway, when/if AIs start to create the narrative to the extent they can author our religion, they already have full control. At least if some AI decides to become a God with us as worshippers, at least we stay around a bit longer.

Probably a better outcome than if it decides that it has better uses for the atoms in our bodies.

In fact, this may even be a solution to the alignment problem. An all-powerful but relatively harmless (non-mutating, non-evolving, non-reproducing) single AI that creates and enforces a religion that prevents us from creating dangerous AI's, weapons of mass destruction, gray goo or destroy the planet, while promoting pro-social behaviour while otherwise leaving us free to do mostly what we want to do.

[ben_w]

Rings a bell: "Thou Shalt Not Travel Into Thine Own Past Lightcone", unfortunately the search results are now overwhelming "NovelAI" and actual physics, so I can't find the name of the story I'm thinking of (and have yet to actually read).

[akomtu]

A distinct property of a machine is determinism. Machines are made of electrons that obey the wave equation: it states that all electrons evolve as one. However we skillfully limit this waveness and squash electrons into super deterministic 01 transistors. We need it because our computing paradigm revolves around determinism. A more general theory that works with probabilities and electrons as they are hasn't been developed yet. And here lies the danger: if a super deterministic AI becomes the ruler of our society, it will prevent further growth. The same phenomenon on individual level is called arrested development: when the individual over-develops a minor skill, becomes obsessed with it and it blinds him from exploring anything else.

A still possible future is when science makes a breakthru in quantum computing and finds a way for humans to steer AI with their minds: it would be the neuralink in reverse. This would force science to research the true nature of that connection which will help to avert the AI doom.

[trashtester]

I'm not aware of any evidence that human brains involve any quantum computing. In fact, I'm pretty sure there's enough noise in the brain that almost instant wavefunction collapse/decoherence is guaranteed.

Anyway, a combination of noisy data entering from the outside world and chaos theory/mathmatics in the equations of most computer systems, I don't see any risk that AI's should get stuck in an infinitely repeating pattern.