[nradov]

Stating something more times doesn't make it true. Everything you've written is pure speculation, and alarmist at that. There's no proof that AGI is even possible, and if it is possible there's no proof that it will end humanity.

[995533]

Clearly, AGI-level intelligence is possible, because human brains exist.

So unless you pose that a function has to rely on its materialization (there is something untouchably magic about biological neural networks, and intelligence is not multiple realizable), it should be possible to functionally model intelligence. Nature shows the way.

AGI will likely obsolete humanity. Either depricate it, or consume it (make us part of the Borg collective). Heck, even a relatively dumb autonomous atom bomb or computer virus may be enough to wipe humanity from the face of the earth.

[nradov]

It's not at all clear that AGI is technically feasible. Human brains exist but we have only a shallow understanding of how they work.

Even if we assume for the sake of argument that AGI is possible, there's no scientific basis to assume that will make humanity obsolete. For all we know there could be fundamental limits on cognition. A hypothetical AGI might be no smarter than humans, or might be unable to leverage it's intelligence in ways that impact us.

Nuclear weapons and malware can cause damage but there's no conceivable scenario where they actually make us extinct.

[995533]

Something can be possible, while still technically not feasible.

I agree our knowledge currently is lacking, but see no reasons why this will never catch up.

There are fundamental limits on cognition. For one our universe is limited by the amount of computing energy available. Plenty of problems can be fully solved, to where it does not matter if you are increasingly more intelligent (beyond a certain point, two AGI's will always draw at chess). Another limit is practical: the AGI needs to communicate with humans (if we manage to keep control of it), so it may need to dumb down so we can understand it.

Even an AGI as smart as the smartest human will greatly outrun us: it can duplicate and focus on many things in parallel. Then the improved bandwith between AGI's will do the rest (humans are stuck with letters and formulas and coffee breaks).

Manually deployed atom bombs and malware can already wreck us. No difference with autonomous (cyber)weapons.

[nuguy]

Even anti-alarmists don’t ask for proof that AGI is possible. Obviously it is possible. Speculation is the best you get because nobody is going to be able to prove anything. We haven’t proven global warming is caused by humans but it’s still worth it to be proactive about greenhouse gasses. This is because when something is extremely dangerous, you don’t wait around for someone to finish proving it beyond any shadow of a doubt. You probably also think that god exists because nobody can prove otherwise?

And what does alarmist even mean? Do you call global warming advocates alarmists? It’s such an annoying, nonsense word that boils down to name-calling really. Discuss the merits of my actual argument. If you think my speculation is wrong, point out a flaw in the chain of logic that leads to my conclusion. Don’t just wave your hand and say that “you can’t prove it” like some evangelical christian talking about god or global warming. Seriously infuriating when there is so much at stake.

[nradov]

I ask for proof that AGI is possible. Show me a computer as smart as a lab mouse and then I'll take your concerns seriously.

The analogy to anthropomorphic global climate change is a non sequitur. Climatologists have created falsifiable theories which make testable predictions.

And you really have no clue about my personal religious beliefs. Calm down and take a seat.

[mindcrime]

I ask for proof that AGI is possible. Show me a computer as smart as a lab mouse and then I'll take your concerns seriously.

I would argue that, unless you can show why AGI is not - in principle - possible, that the null hypothesis would be that it is possible. Unless we veer off into some weird mysticism, it seems that the human brain turns energy and matter into intelligence somehow, operating according to the physical laws of the universe... why shouldn't it be possible to build something else that does the same?

[nradov]

You can't prove a negative. At this point we don't even know what the principles are.

If you're unwilling to provide me with a prototype equivalent to a rodent mind then I'll settle for a fully developed theory of human cognition. Let me know when you've got one. At least that would give researchers some guidelines to know whether they're making forward progress toward AGI.

[nuguy]

I never said anything about your religious beliefs, only that your need for hard proof one way or the other is similar to the need displayed by people who want to believe in god and who dont want to believe in global warming. there are many such examples but the bottom line is that demanding me to show you a sentient computer right now is not reasonable. We have the ability to reason about things without developing formal, mathematic proof about it. And in this case, there is no possibility that anyone could ever prove that AI is possible or impossible. Nobody can prove what it will do. If accept nothing less than hard proof then you are opting out of survival in basically any situation whatsoever. Sometimes logical reasoning is all you have and this is a case of that.

Yeah and before the measurements were done, before enough time had elapsed for meaningful change to be measured, all there was was people like me screaming at people like you, trying to make you see. When ai comes you’ll have your proof but it will be too late.

[nradov]

No that's not how it works. Currently a belief in the inevitable arrival of AGI is a secular religious faith with no basis in hard science. We have plenty of real problems to worry about (like anthropomorphic global climate change) before we waste time making public policy to prevent something that may never happen anyway. Your concerns are silly, akin to worrying about an alien invasion when we don't even know if extraterrestrial life exists.

[nuguy]

You are an outlier. Even Hassabis and Hinton believe that agi is inevitable. I’ve never met anyone who thought otherwise. The only disagreement is about the timescale and the result of its existence. If nothing I’ve said can convince you that ai is possible, then i would just ask you to consider that the human brain exists and that we will eventually figure out how it works.