[constantcrying]

Who is this "we"? Supposing a single person disagrees and decides to give AI more control and gains a very significant advantage by that, what then?

I think people keep forgetting that "Selection" can be excessively cruel.

[hamburga]

A single person acting in isolation (no friends, no colleagues, no customers) has very little agency. While theoretically a single person could release smallpox back into civilization, we have collectively selected it out very effectively.

[constantcrying]

The question is only relevant if AI is a significant force amplifier. If it is not, it is an unthreatening tool, whose usage should be largely unrestricted, this is the case now.

If that ever changes then there is the question what to do with it and at a certain level of power an individual decision would have impact, if sufficiently amplified.

[hamburga]

AI is definitely a significant force multiplier in many areas. Still, an individual in total isolation has limited agency.

[daveguy]

You say that like AI has some sort of autonomy. It doesn't.

[selfhoster11]

It doesn't, by default. All it takes is a capable enough model without rails, and a single user instructing it to act autonomously as its primary goal.

[daveguy]

The point is, AI is not able to act capably or autonomously. If someone tells it to "act autonomously" it'll either sit like a lump or flail until the power runs out.

[hamburga]

All it takes for somebody to nuke Atlanta is an atom bomb and an airplane and somebody willing to fly the plane.

I’m being facetious but there ARE ways to decide/act as a society and as subgroups within society that we want to disallow and punish and select out qualities of AIs that we think are unethical.

[constantcrying]

Doesn't matter.