[0F]

AGI will make many things possible that we’re not possible before. It will invent new things that wildly alter the reality of the world. The point is that you cannot predict what will happen; a million things could happen. But the good outcomes can be counted on one hand. The probability that we get a good outcome is one in millions, probably less. Obviously that is an incredibly stupid thing to do!

[nightfly]

That a lot of words without saying anything at all. What are you afraid they will do?

[0F]

If you can’t draw a conclusion from that then you simply don’t get it.

AGI will, as I said, make many things possible. It will be possible to keep a human being alive forever and suspend them in a state of pure pain for eternity. It will be possible to extract memories and information from the human brain. It will be possible to control human behavior with invasive brain surgery. Imagine every grotesque thing that you can do to a human being — it’s on the table for AGI enabled governments in the early days if that’s how it plays out. And that’s just one slice in the infinite pie of grotesque possibilities. You can’t just opt out of these things if they provide a material advantage to your country/meta organism.

[nightfly]

I've got an imagination, but you're letting yours run wild. AGI doesn't mean immediate sci-fi level medical technology, and it certainly doesn't mean human brain copying and transfer of consciousness to some artificial storage device.

Biological brains are incredibly complex and if that level of complexity is needed to host intelligence I think we're a long ways off from building any true AGI, even if we pumped every bit of engineering skill our society has at it. And even if we _can_ build AGI I doubt we'd have the resources to host more than a handful of them for years to decades after their creation or that their intelligence would be anywhere near small mammal level.

[0F]

If that level of complexity is needed. If. Of course it’s not.

[nightfly]

What makes you so confident about that?

[postalrat]

I'd be afraid it's going to do something you can't hope to predict.