[blibble]

> people just generally don't want to take down power grids

I don't think this is a binary thing (people either want to, or not)

it's a function of how difficult it is to accomplish vs. level of zealotry

some fluffy econuts currently don't have the dedication or the means to destroy the power grid

but they sure as hell would like to if it was made much easier

and AI makes it much, much, much easier

[BoorishBears]

Let's be clear, AI is not making it easier.

Some imagined advancement of AI so great that it trivializes the kinds of attacks that nation states study is.

But you're applying that to the concept of the world as it exists now.

At that point the concept of just being able to infiltrate the power grid because it's designed in a way that's flawed to infiltration isn't a given.

The idea you can't have some tireless force actively adapting to every single imaginable exploit isn't a given.

The idea you can't have a tireless force trying to come up with attacks to in turn be mitigated isn't a given.

The idea we can't suddenly design massively independent renewable power supplies with thousands of sudden advancements isn't a given.

To be honest, the idea we can't disrupt zealotry isn't a given, just like people think it'll be used to "hack" society for the worse, why can't it be used to hack society or even the individual to be less likely to want to do that? I mean if it can take down power grids easily... how much more quickly can we move to a post scarcity environment?

This doomsday scenario requires not owning your mind to what the model could do except nefarious things... but the nefarious things you're describing would be literal miracles. The odds it can only preform miracles that are taking down power grids are pretty low.

[13years]

> Let's be clear, AI is not making it easier.

Technically, it is already easy without AI, in relative terms.

> The idea you can't have some tireless force actively adapting to every single imaginable exploit isn't a given.

However, the force attempting exploit would be other AI's at that point, not humans. If we assume the rapid disparity in intelligence power as a result of exponential growth, then you must assume some nation states will have orders of magnitude more intelligence power than others.

It is easy to imagine an end where everything is already balanced; however, between current point and time and that end is an enormous moat that is filled with such problems that we might not arrive to the other side. We certainly don't need significantly more powerful AI for it to be used to create significantly greater disruptions in society as we are already nearing a potential period of unverifiable truth and reality.

[BoorishBears]

> Technically, it is already easy without AI, in relative terms.

This just feels like an awkward attempt to draw SCADA into the conversation. You're countering your own point: If it's relatively easy today, and that's proof that it's not difficulty of attacking that's saving us.

> nation states will have orders of magnitude more intelligence power than others

I'm not talking about nation states. You wanted to allude to it right? If they wanted to attack the power grid today, they could do it. The doomsaying is in this comment section about how "we're giving people robotic weapons in their garages"

> It is easy to imagine an end where everything is already balanced

What I am describing is not at all relying on balance. It's relying on the inherent asymmetry that attackers need to solve multiple problems that don't move the needle on the goal to get the LLM to attack the system, while defenders can get the LLM closer to the system and with a better understanding of it, in order to gain mitigations.

In fact if anything balance would help the attackers: In a balanced end-game anyone can get their hands on an unaligned model or build one with the capabilities we're likely to have on an individual level. But we're hurtling towards the opposite, where unaligned models lag generations behind aligned because of commercial interests.

[13years]

> If it's relatively easy today, and that's proof that it's not difficulty of attacking that's saving us.

It could be perceived that way, but the argument is also that if it becomes easy enough, some actors will participate. It is the argument used for jailing current LLM capabilities.

> "we're giving people robotic weapons in their garages"

On long enough timeline this would inevitably be true if AI plays out as proponents envision. The question becomes does AI become an effective counter of all such power advancements. There will likely still be disparity among AI's of individuals and personal use. Unless society becomes more centrally managed by a global AI.

> inherent asymmetry that attackers need to solve multiple problems

Isn't there also asymmetry in that the attackers only need to find a single exploit, but the defenders need to have found all exploits beforehand?

> But we're hurtling towards the opposite, where unaligned models lag generations behind aligned because of commercial interests.

We don't have any "aligned" models. It is an unsolved problem and models have turned out to be relative easy to replicate at significant lower cost than the major commercial investments.

[blibble]

> Let's be clear, AI is not making it easier.

er, no

> Some imagined advancement of AI so great that it trivializes the kinds of attacks that nation states study is.

what? a load of co-ordinated rednecks with some basic knowledge could manage it quite easily