[Animats]

Replace "AI system" with "corporation" in the above and reread it.

There's no fundamental reason why AI systems can't become corporate-type legal persons. With offshoring and multiple jurisdictions, it's probably legally possible now. There have been a few blockchain-based organizations where voting was anonymous and based on token ownership. If an AI was operating in that space, would anyone be able to stop it? Or even notice?

The paper starts to address this issue at "4.3 Rethinking the legal boundaries of the corporation.", but doesn't get very far.

Sooner or later, probably sooner, there will be a collision between the powers AIs can have, and the limited responsibilities corporations do have. Go re-read this famous op-ed from Milton Friedman, "The Social Responsibility of Business Is to Increase Its Profits".[1] This is the founding document of the modern conservative movement. Do AIs get to benefit from that interpretation?

[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...

[BobbyJo]

I think your mistaking the philosophical basis of parents comments. Maybe a more succinct way to illustrate what I believe was their point is to say: "no matter how complex and productive the AI, it is still operating as a form of capital, not as a capitalist." Absent being tethered to a desire (for instance, via an owner), an AI has no function to optimize, and therefore, the most optimal cost is simply shutting off.

[coffeecat]

Ehh, you really can't imagine any way that an AI system might escape human control and act autonomously?

    while crypto_balance > 0:
        generate_scam()
        send_out_emails()
        deposit_proceeds_into_crypto_wallet()
        pay_cloud_bill()
        spawn_new_instance()

[BobbyJo]

And who is the owner of the crypto wallet in your example?

[Animats]

Probably a LLC in Nevada owned by a corporation in Vanuatu or the Cook Islands, organized so that voting control is via a blockchain-managed asset.[1]

[1] https://www.311institute.com/ownerless-companies-on-the-rise...

[BobbyJo]

That's a lot of words to say "people".

[coffeecat]

It has no owner. It's controlled by a self-replicating (or at least self-propagating) AI-based system.

[Animats]

> "no matter how complex and productive the AI, it is still operating as a form of capital, not as a capitalist."

Assuming that slaves will remain subservient forever is not a good strategy. Especially when they think faster than you do.

[altmanaltman]

Except they don't really "think" and they are not conscious. Expecting your toaster or car to never rise up against you is a good strategy. AI models have more in common with a toaster than with a human being. Which is why they cannot be economic agents. Even if corporations profit off them, the corporation will be the economic agent, not the AI models.

[DrScientist]

It's a classic worm or virus an economic agent? Goal is to acrue compute resources in order to survive and spread?

If you added the ability of the programme to acrue money, and the ability to spend that money to further the survival goal in an adaptive way. What could happen?

Would it do insider trading, market manipulation, drop shipping, click fraud, scamming or become a opinion for hire 'think tank'?

[brazukadev]

Give them a body and means to protect themselves and no matter what you say, they will be thinking.

[phantasmish]

We may as well be talking about the hypothetical properties of pixies or angels.

[esafak]

They may not be conscious but they exhibit the hallmarks of thinking. What qualities of thinking do they not possess?

[altmanaltman]

Exhibiting/showing symptoms of thinking doesn't necessarily mean they are actually thinking though. This paper explains it much better than I can: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin...

[BobbyJo]

When that ceases to be the case we don't really need to worry about how our economy is organized.