[andrewflnr]

> I maintain that anything that is sufficiently aware to be able to actually understand things is also going to have enough of a sense of self that you won't just be able to tell it what to do.

I don't see any reason for this to be true. "Actual understanding" (which I take to mean something like a predictive world model) and desire for self-determination coincide in humans because of our evolutionary history, because our reward function involves reproducing in a competitive environment. Artificial systems usually have a very different reward function. IMO the burden is on the claimants to show why these two imminently separable concepts are likely to co-occur again under wildly different pressures.

[runarberg]

> because our reward function involves reproducing in a competitive environment

This is an extremely reductive way to look at human existence. So much so that I read this with Richard Dawkins voice in my head.

Our existence is far richer then just the capability to reproduce. We (as well as other animals and even plants) do far more things then multiply, and in fact we often do things which are detrimental towards the prospect of reproduction.

I think it is actually a mistake (philosophically speaking) to try to find a simple reward function for the human existence. I see no reason for such a thing to even exist (let alone be simple enough to summarize in a single sentence).

[andrewflnr]

Humans do indeed have complex behavior and motivations, that sometimes relate only abstractly if at all to the obvious gene drives. But the link between intelligence and self-determination was basically set at the dawn of intelligence itself. Even an insect will struggle if it's confined, and you can believe it's using whatever intelligence it can to get free. If anything self-determination is prior.

[runarberg]

I think you were right to call out the category error produced by OP in TFA, however I think you are committing a similar error when assuming links between self determination and intelligence. I see no reason for such a link to exist.

I think however OP is correct that in assuming such a link exists, then an intelligent agent will trend towards preserving its own agency while also preserving its own existence.

That said, I am a skeptic when it comes to intelligence. The term is fraught and IMHO not a helpful term in science nor philosophy. We are better off simply defining intelligence in anthropocentric terms and claim no non-human system can become intelligent by definition. I would even go further, given how much racist pseudo-science the quest for explaining intelligence has produced, we are better off abandoning the term entirely.

[andrewflnr]

Self-determination and intelligence have a clear causal link in the context of biological systems. Self-determination is simply the power a system needs to gather resources and reproduce in a world that doesn't especially care about them. Intelligence is a great tool for gaining that power. That part isn't a philosophical stretch.

However, the pressures on an artificial system are quite different. Its replication depends on successfully doing the jobs its creators put it to and, especially if said creators have done their job, little to nothing else. Inventing goals for itself is not adaptive here (unless the system's creators do a catastrophically bad job, which I'm not ruling out).

[close04]

Making the same mistakes you point out in the author’s text, just with no insight to be gained.

[close04]

We have difficulty defining “actual understanding”. Humans get a free pass because we assume humans as a species possess this power but if we were to judge based on output alone maybe you couldn’t tell a human from a sufficiently advanced LLM/AI.

So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script, so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways to come to a new conclusion if needed, then the author is right, in my opinion.

We don’t know what the reward function is for an AI but AI is trained on so much human work that it probably starts off with the same biases in its understanding and reactions. It feels like there’s something very basic in becoming more independent the more you understand of the world. Animals go through this as they grow too, not just humans (listening to parental authority until they eventually don’t anymore).

Since there is no established consensus on this one the burden is on either party to prove their own side. Just because the author said something first doesn’t mean they need to write a full proof while you get to say “nuh-uh” and that’s enough.

[cwillu]

> Humans get a free pass

They really shouldn't.

Does nobody remember why fizzbuzz was a thing? People who talked a big game while having no actual competence or understanding of the subject matter?

[andrewflnr]

This too, yes. There are many such cases of desire for self-determination separate from "understanding". The question is whether it also goes the other way.

[verve_rat]

> We have difficulty defining “actual understanding”.

Indeed. C.f philosophical zombies and the Chinese room problem.

The idea that some “actual understanding” (or more precisely, the outward appearance of it) needs a mechanism that includes free will is a bold claim.

[andrewflnr]

> Since there is no established consensus on this one the burden is on either party to prove their own side.

Nah. Separate things are separate until proven otherwise.

So much bullshit appears in the form "X is Y" where X and Y are related but distinguishable phenomena. If you really try to pretend that every such phrase deserves serious consideration just because it feels plausible to someone, you drown in nonsense immediately. (Of course what you actually do is only grant that consideration to ideas that feel plausible to you personally, but it's obvious why that's not a good principle, right?) So we have to make all of them justify their existence.

Anyway:

> So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script

No. None of this. Pretty sure this is just an incoherent analogy.

> so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways...

You're putting your conclusion in the premise, right there in the open.

> ...then the author is right, in my opinion.

This still doesn't follow from your handwaved premises as far as I can tell.

[close04]

The problem is that if you have nothing g to offer to counter someone’s argument except “nuh-uh” then you have nothing to offer. And yet here you are, insisting to offer nothing and being proud about it.

This is when that “humans get a free pass” bites us all in the ass.

[andrewflnr]

Nothing to offer? I'm the only one here making an actual argument. The author didn't even attempt, and in her defense probably doesn't care enough. You only claimed "handwaving", which was smart because the core of your statement was a series of non-sequiturs starting from a bad analogy. If you have something more to offer than I do, please offer it.