[ACCount37]

"Alive" is irrelevant. Bullshit like "alive", "sentient" or "conscious" doesn't decide much, at the day's end. It just distracts from the thing that matters.

And the thing that matters is: capability.

Even today's AIs are capable of autonomous goal-oriented agentic behavior - and growing more and more so with every release.

At sufficient capability, AI stops being "a technology we use", and becomes a force in itself - not unlike humankind. Because intelligence is very powerful - it's what allows humans to dominate the world. A world where human intelligence has a peer is a world where human control is contested. And beyond that?

You might end up being more outclassed by AI than a toaster is outclassed by you.

[anon7725]

> "Alive" is irrelevant

Alive is always relevant. Our moral consideration must place beings that are alive or processes that support the life on this planet above things which are not alive.

In a just world, it must be adapted to us. It’s merely a file sitting on disk with numbers in it. And to put a fine point on it: the ambitions of the wealthy people who make it must be adapted to us.

[falcor84]

Then according to your argument, the moral consideration only depends on the moral agent, right? So if tomorrow AIs decide that they are alive, it would be morally incumbent upon them to adapt the world to support themselves, correct?

[anon7725]

> So if tomorrow AIs decide that they are alive, it would be morally incumbent upon them to adapt the world to support themselves, correct?

The only system of morality that applies to LLMs is their effect on us and the planet. Don’t anthropomorphize a matrix.

[lukan]

You seem to have the assumption, that humans MUST have the moral to preserve primarily HUMANS?

I don't think that is universal true. Personally I my ethics revolve around how to preserve and spread life in general. I happen to be human and we are the most capable and sentient species so that means creating and maintaining conditions specifically for humans, but if another species comes that is better suited and acts better then it would be selfish to insists on us.

Having said that, AI to me is a tool, not a species and I don't see how it can be ever be sentient. But I also don't know how consciousness works. So I am open for the questions of the future in the future - and now rather focus on the practical implications that AI will have on our current society.

[falcor84]

I'm not anthropomorphizing them, I'm animifying.

[w4der]

For AI to decide that they are alive and moral agents would require them to be able to take moral responsibility for their actions. As long as the labs that run them deny that responsibility, or an AI can´t convincingly demonstrate it can take moral responsibility on its own, they are not moral agents.

[baxtr]

What does AI want? Without ambitions you have to be told what to do.

[ACCount37]

What makes you think that AI can't any ambitions?

Today's systems are mostly good at following instructions. But push them far enough, and you already get weirdness like alignment faking, instrumental self-preservation and more. We know because we've seen it in lab settings, while probing for extreme failure modes on purpose - but the world is large and strange enough that edge cases like that are liable to surface naturally.

The reason why none of this has exploded in our faces isn't that today's AIs never want to do weird and dangerous things. We know they do, at times. It's that today's AIs are incapable of pulling them off when they try.

Capability is the thing that matters, at the day's end.

[RandomLensman]

Current AI systems have started to do things without ever being prompted for anything at all?

[falcor84]

Not as far as I know, but I have personally seen my coding agents take on a prompt like "See if you can port this project to typescript", and work for hours, defining a myriad of subgoals and continuously summoning and managing subagents while developing ad-hoc tools and skills.

There is to the best of my knowledge no fundamental limitation to having an agent/claw go on like this for 80 years with a prompt like "live your life to the fullest".

[TeMPOraL]

All the time, yes. But you have to keep two things separate in your thinking:

- Prompted as in prompt made of tokens -> for LLMs, tokens double as a clock signal. Time only flows when tokens are pushed through them.

- Prompted as in specific request placed in the stream of tokens -> Yeah, they do that all the time whether it's getting into infinite loops of repeating same pattern, or suddenly deciding to do things based on inputs they normally ignored.

Also don't forget that everything is a "prompt" for LLM. All input tokens end up in the same place.

[RandomLensman]

So without a token pushed into them they do something? Not sure I understand...

In the current UIs is there a lot of suppression then as I have not seen things start on their own?

I meant an LLM doing something without any external prompt at all. Not doing something different etc but rather do something without a token/prompt ever flowing to it.

[ACCount37]

Category error. You might as well say "I've never seen a human do anything without being alive, so humans must be very easy to use safely".

Today's AI harnesses just keep feeding AI tokens - and can, in principle, do so indefinitely. "Streaming LLM" is not mainstream but not unknown either. "Automatic context compaction" is somewhat similar in what it does, and very common. "Keeping the lights on" isn't hard. And you have to turn on those lights to have an AI do anything at all.

At the same time: that type of harness often provides a "tool call" interface. Plenty of tools that end up attached to LLMs can do things like run arbitrary code, spawn instances of the same AI, and more.

That's frankly enough. An AI that can spawn more copies of itself can, in principle, just keep itself running indefinitely - even in a non-streaming no compaction harness. An AI that can run arbitrary code on a system can, in principle, do anything a human using that system could do.

The thing that truly limits what an AI can do is the AI itself. Practical AI safety relies on AI being either too weak or too well behaved to cause major issues.

[baq]

You’d need to train a model to be able to do that.

We shouldn’t be doing that as a civilization in case it works.

[everyday7732]

If they are set up that way yes they can continue indefinitely. Why would they not be able to do this? Take a look at the experiment called the AI village. https://theaidigest.org/village. I think it pauses sometimes because AI is expensive but could be running continuously with a larger budget

[RandomLensman]

That village was never started, so to speak?

[alecco]

DNNs/LLMs can only predict next tokens based on training data. They often make big direction mistakes as they are particularly bad at common sense. Kind of like the Paperclip Maximizer scenario. They need a human with deep knowledge to drive them and to catch them when they go off the rails.

[ACCount37]

"Next token prediction" isn't a system. It's an interface a system uses. Nothing precludes an arbitrarily simple or complex behavior from producing a token logit.

And with what we know of LLMs? Autoregressive transformers are Turing complete in theory, and we are yet to find anything that LLMs are "fundamentally incapable" of in practice. Even continuous learning is already approximated with in-context learning - both allow a system to learn from prior experience, both have practical limits on how far they go. That's what powers "trial and error" in today's agentic LLMs.

"LLMs can only predict next tokens based on training data" is comforting but misleading. It just isn't the saving grace you want it to be. It describes an interface, not a ceiling. And if there is some sort of fundamental "capability ceiling" that LLMs are heading towards, we are yet to see it. We know plenty of things LLMs are bad at, but they keep getting less bad at them release to release.

If there is none, then, simply improving over the current recipes iteratively might yield systems that only "need a human" in the same way you "need" to have a boss. Maybe less so.

[Lerc]

>DNNs/LLMs can only predict next tokens based on training data.

How do they decide between using 'a' or 'an'?

[falcor84]

I don't get the argument; how do you decide between using 'a' or 'an'?

[Lerc]

You use 'an' when the word that comes after it begins with a vowel.

[alecco]

They pick random top-k next token based on their amazing 4chan/reddit training data, duh.

[Lerc]

So you you think a model if asked.

"There is an animal very similar to a crocodile but I cannot remember it's name"

and the model responds with

"I believe the animal you are thinking of might be " ("a" / "an")

Are you saying that it would pick the result fairly randomly and then based on it's choice pick an animal that starts with a consonant or a vowel?

[PowerElectronix]

Human intelligence is going to be peerless for centuries if this AI is the one supposed to be the peer.

[munksbeer]

AI currently surpasses humans at many tasks, and the list grows every month. My guess is that within 20 years (as a hedging my bets 95% confidence level) AI will surpass our intelligence for almost everything.

The arguments over whether it truly "thinks" or "reasons" will be superfluous. They simply won't matter.