[0F]

I’m worried about AGI.

I will make everything very simple. All of AI industry and research boils down to one thing: mining algorithm space. We set up programs that search algorithm space automatically until they find one that demonstrates desirable behavior. And all the progress in recent years boils down to this: we have hit veins of intelligence in algorithm space that have exceeded what we thought possible.

If you think you know what’s going to happen next because leading edge algorithm X has properties Y and limitations Z then you are lost. We are mining algorithm space and we keep striking veins. Money is being poured into mining and it is a fact that we will keep hitting veins.

General intelligence is probably a broad category. If you were to define it mathematically, there is probably more than one mathematical kernel that can underpin a generally intelligent algorithm. So the number of algorithms that are generally intelligent are probably much higher than intuition would lead you to believe. Eventually we will strike a GI vein.

Again, if you think AGI will do X or do Y then you are lost. It will only do one thing: it will change the economic equation of life on this planet drastically and permanently. It will be the worst thing that has ever happened in terms of human well-being.

If we took control of the fabs and enacted limits on processor feature size we could slow the mining enough to find a final solution. It sounds silly until you realize that it’s truly the world at stake.

[AgentME]

Luddites get panned today as just being reflexively anti-technology, but they were acting in their self-interest: they were the textile workers whose expertise and careers were threatened by machines. They lost their livelihoods.

It's possible there were alternative outcomes where everyone won. A more fair society could have found a way to make sure the replaced textile workers benefited from the machines that replaced them.

If we make machines that replace all of our economic roles, then it's possible we could make it so we benefit too, but with AGI it's not nearly enough to just choose to use it that way. AGI will not be aligned with our values if we don't make it that way. If we solve the alignment problem, then we would be able to make AGI that shares our values. AGI could solve our problems, or uplift us to its level, if we accomplish this. But this is not the default outcome if we don't solve the alignment problem.

[mkaic]

Yup, I agree. It's inevitable and it's scary but do keep in mind that there's no reason to believe it couldn't also be a positive thing. We have no idea what the long lasting impacts of AGI will be on our society, but we know that they will be massive. It might wipe us out, or it might aide us in building the perfect utopia.

While I am worried about it, I also recognize that worrying about it doesn't really do anyone any good -- the best thing I can do is to keep growing my skills as a tech person so that I can be as prepared as is reasonably possible when it eventually does arrive. Heck, maybe I can actually contribute to helping align it positively with humanity, I don't know. The point is that there's no point in stressing too much about it.

[ruined]

we've been unable to design human-powered systems that reliably value human well-being. hell, humans don't even reliably value human well-being. why do you think any agi or agi-based system we design might be different?

one would expect that initially, agi will simply act in the existing role we have already established for artificial persons - the corporation, notably uninterested in human well-being.

[psyc]

We'd better hope the first one is created by a reclusive mad scientist with a heart of gold, who forgave everyone who laughed at him.

[kovek]

I think it's good to do the following. Assume that jobs are taken over by AGI, one by one. Which job will be left last? Which job would I want, given this change?

[lgas]

I'm not sure I want to be the last human working.

[nightfly]

What are you afraid it's going to do?

[0F]

AGI will make many things possible that we’re not possible before. It will invent new things that wildly alter the reality of the world. The point is that you cannot predict what will happen; a million things could happen. But the good outcomes can be counted on one hand. The probability that we get a good outcome is one in millions, probably less. Obviously that is an incredibly stupid thing to do!

[nightfly]

That a lot of words without saying anything at all. What are you afraid they will do?

[0F]

If you can’t draw a conclusion from that then you simply don’t get it.

AGI will, as I said, make many things possible. It will be possible to keep a human being alive forever and suspend them in a state of pure pain for eternity. It will be possible to extract memories and information from the human brain. It will be possible to control human behavior with invasive brain surgery. Imagine every grotesque thing that you can do to a human being — it’s on the table for AGI enabled governments in the early days if that’s how it plays out. And that’s just one slice in the infinite pie of grotesque possibilities. You can’t just opt out of these things if they provide a material advantage to your country/meta organism.

[nightfly]

I've got an imagination, but you're letting yours run wild. AGI doesn't mean immediate sci-fi level medical technology, and it certainly doesn't mean human brain copying and transfer of consciousness to some artificial storage device.

Biological brains are incredibly complex and if that level of complexity is needed to host intelligence I think we're a long ways off from building any true AGI, even if we pumped every bit of engineering skill our society has at it. And even if we _can_ build AGI I doubt we'd have the resources to host more than a handful of them for years to decades after their creation or that their intelligence would be anywhere near small mammal level.

[0F]

If that level of complexity is needed. If. Of course it’s not.

[postalrat]

I'd be afraid it's going to do something you can't hope to predict.

[AgentME]

Anything with much greater capabilities than us that would find our resources useful and that doesn't explicitly have our best values at heart would be a danger to us. European settlers were not good news for the Native Americans. Early humans were not good news for many great apes.

(It might be tempting to imagine this from a higher-level perspective and assert that each of these successors were somehow more interesting or had more fulfilling lives than what they replaced, and so therefore these successions might be ultimately good in the long run, but that's not meant to be part of this analogy. We won't have some genetic lineage with AGI; it will probably be more alien to our mind's design than actual aliens produced by evolution like us would be. The AGI might not build a world replacing ours where many of its kind have lives we would find fulfilling or interesting. It might care just about making the world completely predictable and safe, empty of outside threats and life, and hibernating in a place of safety. Plenty of animals live solitarily and might do this if they were naively uplifted into superintelligence. Humans and social animals instinctually believe in the goodness of friendship and society because evolution saw we were good at surviving in groups and molded our instincts to encourage us to do that. AGIs won't necessarily have that if we don't design that into them (or set up the process that designs them to do that).)

[DonBarredora]

Do yourself a favour and lay down the science fiction for a while.