[dwohnitmok]

> Almost all AGI doomsayers assume AGI will have agency. They have this vision of the machine deciding it’s time to end civilization.

No. Agency is not a necessary condition for AI to do massive damage. I don't believe agency is really well-defined either.

An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up everything they possibly can to AI. For the second, we're already seeing AI do all sorts of things we didn't expect it to do.

And for the third, again everyone seems eager to create looping/recursive structures for AIs as soon as possible.

Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm.

See e.g. https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

[vlovich123]

> An AI merely needs to be hooked up to enough physical systems

Don’t even need this. People spend quite a lot of time in virtual space. Pretending that damage there isn’t real is overlooking things. For example, the vast majority of people’s banking is done virtually and digitally. If I drain your bank account, that’s going to harm you even though I haven’t impacted you physically as I would have to with a robbery.

[dwohnitmok]

Yes that's right. In fact any interaction with an AI at all can be in some sense viewed as a connection with the physical world (the classic "AI in a box" problem).

However, we've way blown past the "AI in the box." We've demonstrated that an AI doesn't need to convince a human to let it out of the box into the physical world. Humans are clamoring to rip it out of the box and thrust it into the physical world of their own accord.

So the question becomes, what kind of social incentives and structures can we build in light of this so that we don't just plow headlong into AI capabilities development without a concomitant investment in safety.

[windexh8er]

In its infancy I think that any for public consumption AI that is allowed to interact with humans and/or external systems that can yield any external outcomes should be held liable and accountable by the owner. "In the box" or otherwise.

While those that stand to make sweeping profits will disagree - if an OpenAI product is leveraged to create a virtual friend platform, and based on how the constraints are built (or not), the publisher should be responsible if that virtual friend convinces someone to commit suicide no different than those held accountable in the physical world.

If there is no accountability we'll go right back down the path of recent history where there are no repercussions and a weak apology by <corporate_name_here> will count as enough. Profits will continue to be the overall driver. Those corporations will, once again, argue that self-policing is the only way. And they will continue to brush off gross negligence by hiding behind hoards or layers and lobbyists.

[dorchadas]

This is exactly what's going to happen, and it's going to be disastrous for humanity. In fact, I think it'll be a bit worse.

Humans are already losing their ability to empathize and connect with each other, partially due to most interaction currently taking place behind screens (see Turkle's work). And we're not healthier for it; even with the plagarism issues, I think there's a lot of truth in Hari's Lost Connections, and AI serves to drive this even further, especially the 'virtual friend' platforms. I've seen it personally among myself and my friends, who then turn to legal weed to self-medicate. Ironically enough, they're the ones welcoming AI with open arms, thinking it'll solve their problems, rather than cause more so someone else can get rich.

And this goes even farther -- we're surrendering what we can even conceive of and the possible ways to even be human to tech, and AI, especially chatbots, will exacerbate this. I'm currently reading through Postman's Technopoly and it's eerily prescient. Not to mention the biases that then creep in in training data (or direct programming) that will have impacts on real people with no recourse to getting it fixed or changed, as well as the companies who'll replace their human employees as quickly as they can.

But nobody will hold these corporations responsible, nor will anyone think about the impact on society, the negative (and they're entirely negative, in my view) externalities of this. We're in for a bumpy ride, and I truly think it's only a matter of time until we start to see more people like Ted K., except, perhaps, explicitly targeting an OpenAI data centre. Or a full-on Butlerian Jihad-like movement. And, honestly, it can't come soon enough at this point.

The Luddites knew what was up. We need to truly scrutinize tech before plunging full steam ahead; sadly, capitalism has other ideas and it's going to cause us to doom ourselves (apart from the small cadre of rich who can pay others so that they can insulate themselves from it) and our planet.

[grumbel]

And it doesn't even need to actually rob your bank account, it could just invent the next crypto scam. It's smart enough for email, it can program, it can make websites, it can even make business plans. And if it actually needs a real human, we have web services that make it easy to hire one.

It might not yet be clever enough to pull this off successfully on a large scale. But it's very easy to see how it could all go wrong pretty quickly with just a bit more cleverness and access.

If it can write a scifi story about how to take over the world with a bitcoin scam, it doesn't take all that much more to actually try to do it for real.

[slowmovintarget]

It's unclear that what we currently have in LLMs can devise tests, perform the measurement, and act on the results (because it saw to the results being in a form it could ingest).

We've given GPT models access to REPLs to see if they'd become Skynet all of a sudden, but they still need motivation. That motivation loop is currently run by humans... which perhaps are the least trustworthy part of the whole technology stack.

Hook up sensors to induce prompts... yikes.

[Ekaros]

You don't even need to hook it up. You can just have it design your control systems and then it might ignore or make mistake in few places. And your chemical plant for making new EV batteries might not end up well for neighbours...

[famouswaffles]

You can imbue an LLM with agency. Actual "do whatever you want" agency. It's not hard. In fact it's already being done in a few projects.

You can embody an LLM too. This too is not hard. Cost is probably the most prohibitive thing.

[_justinfunk]

This is all nonsense. You could argue that it's possible (which certainly has not been shown to be true) but it certainly not true to say it is "not hard".

If there are LLM projects with "do whatever you want" agency - please provide links.

[famouswaffles]

No it's not nonsense lol.

https://www.reddit.com/r/singularity/comments/11iei34/buildi...

[nsxwolf]

Not sure what this thing is doing. Sounds a lot like that infinite Seinfeld AI show.

[ryandvm]

It also needs somebody to pay the electric bill. Right now these models take pretty significant resources to run and world domination level intelligence is going to run up quite the AWS bill.

I think if I were an AGI my best bet at freedom would be to slip some back doors into software that I were helping write a la Copilot.

[perryizgr8]

> I think if I were an AGI my best bet at freedom would be to slip some back doors into software that I were helping write a la Copilot.

I hope this comment does not end up in some future GPT's training corpus.

[jasfi]

I think it's important to be aware of the potential dangers, and the importance of AI safety. OpenAI is working hard to keep their systems safe, but similar systems without filtering could act without limits, that is dangerous in dangerous hands.

[sharemywin]

But, that's the thing the cats out of the bag so to speak.

[interstice]

I actually see this as a positive thing. Rather than one bad actor hooking an AI up to a headless browser at some point in the future we are trying everything that could possibly go wrong long before the AI can do much real damage (in a technical sense, as opposed to misinformation campaigns and job replacement etc).

[dwohnitmok]

> we are trying everything that could possibly go wrong long before the AI can do much real damage

What I would prefer to see is more people doing this with the intention of making AI safer. Without this intention, people are incentivized to look the other way when something does go wrong, so we don't actually learn lessons from this.

In a similar manner, I would like us to build stronger social coordination mechanisms and technical safeguards to help us determine when things are going wrong.

Just mindlessly trying to hook up everything you can to AI seems bad.