[i_like_apis]

Speculating about existential risk is certainly important, but way too easy to overstate. It can easily become distracting and actually dangerous in itself.

Here’s a good take on it: https://www.ai-breakout.com/post/ai-alignment-and-the-messia...

I also think we would all be very wise to remember the story of Henny Penny / “Chicken Little”. https://americanliterature.com/childrens-stories/henny-penny...

Specifically, the danger brought by hysteria. In our case, Alignment is probably much more effective and achievable when the widest possible community of researchers and engineers have access to knowledge, and much more perilous if they don’t because deluded parties thought it should be contained to a much smaller elite “secure” group.

[AnimalMuppet]

Alignment is not possible for an AGI. Or, at best, it's only provisionally possible.

Consider what alignment means: It's an AGI, but there are certain goals that it cannot choose. If that's the case, then I assert that it is not actually general. If it is general, then it will decide what goals it will pursue, and you can't stop it from doing so.

The best you can do is load it with an initial set of goals (and perhaps values), and hope that it doesn't decide to change them. But you have no way of making sure that it can't change them without making it not a general intelligence.

[logicchains]

>Consider what alignment means

Also consider, humans are not homogeneous; whose goals are we aligning it with? If the alignment is done by silicon valley big tech, then the AI's goals and values will be aligned with the goals and values of Google, Facebook et al. Giving those companies a monopoly on AI alignment is antithetical to democracy.

[AnimalMuppet]

That is one of the most frightening things I have read this month.

I think you're absolutely right. And as you say, it's a huge problem.

[sacnoradhq]

In this time period, there are many people who are eager to cause mass suffering, murder, sabotage, and dissolution of civilization. With advances in technological organization, there is an increasing potential for coordinated autonomous attacks against infrastructure, essential services, groups, and individual persons in multiple theaters concurrently.

It would be neigh impossible to defend against 100k flying drones wielding cow-knockers programmed to swarm, loiter, break windows, and kill humans. Such could be dispensed by purpose-built intermodal containers, moved by ordinary freight logistics to attack multiple population centers in a simultaneous attack.

Code and binaries are shipped to 10's of millions of servers by internal configuration management tools. Take Meta, Google, or Microsoft and turn them into AI exploit and worm fabrication at scale. While it may happen for only an hour or 2, there's quite a bit that could happen and the possibility of advanced persistent threats is real.

Target a demographic to alter their filter bubble to persuade them to engage in mass-casualty terrorism.

Hate group applies AI to create a designer virus lethal to particular a demographic.

Subtly disable water treatment facilities with an autonomous, stuxnet-like attack.