[f_klem]

> It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".

Subjective experience is what you and only you, different from other beings, experience in and about the world.

> Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?

You can't be sure that machine do not work like our minds or brains. But you cannot say the opposite either, so saying 'machines could think' base on a false assumption (because you cannot say it is true. It doesn't matter if you could be true).

> Linguists are linguists, they don't know about consciousness, they specialists in language.

That's a very narrow understanding of what language is, what linguists do/research, and the contributions made in the field. Linguists are (since already 2 o 3 decades) focusing more and more on psychological/cognitive matters. The intertwined topics of language, mind, body and though has a long way in the western philosophical tradition.

> I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?

Exactly because of this. And is this what I am talking about... the topics you mention here are already settled in the philosophy space, but the AI research space keeps going 'round them...

[ordu]

> Subjective experience is what you and only you, different from other beings, experience in and about the world.

So subjective experience is subjective experience? Do rocks have subjective experience? The main goal of any definition is to say clearly what is doesn't cover. It is not enough to say what is subjective experience, you should say it in a way that excludes everything else. Like what is objective experience? How it differs from subjective one?

Do LLM experience objectively or not experience at all? How can you say?

> But you cannot say the opposite either, so saying 'machines could think' base on a false assumption

I can say that they could think. The thought process implies a measurable product. Yes, there are similar situation with it, it can be hard to say sometimes if we observe a product of a thought or something else. But science has some success with this, like claiming that bees can think and solve problems.

> That's a very narrow understanding of what language is, what linguists do/research, and the contributions made in the field. Linguists are (since already 2 o 3 decades) focusing more and more on psychological/cognitive matters.

I'm telling you, psychologists (who specialize on mind research) do not know what consciousness is and they do not have a definition of a subjective experience (well, if we treat your proposed definition as a valid definition, then we should say they have plenty). And my claim still stands: until you heard about new science "Subjectology" you can be sure that no one knows what subjective experience is. Including linguists.

> the topics you mention here are already settled in the philosophy space

You shouldn't believe that anything can be settled in the philosophy space. Philosophers can think they have settled things, but until science agreed and started empirical studies, it is just philosophers believing that they settled things.

> Exactly because of this. And is this what I am talking about...

Exactly because of this I'm vary of any "settled things". You see, people have all the reasons for motivated reasoning. People are historically very wary of any attempts to extend the group of beings covered by "human rights" or whatever it was actual at their time. People would fight to the death to not include another group of beings to the list of "sentient", "conscious" or what it is today is euphemism to "morally equal to a human". And I do not trust anyone to overcome this deep psychological bias. I do not trust myself, I do not trust philosophers, I do not trust linguists, I do not trust neurologists and psychologists. Well, psychologists are probably the most prepared to it, but I do not trust them still.

Just think about it, lets suppose someone proved beyond any doubt that LLM has conscious experience. What happens then? Try to imagine that counterfactual world. It will be a very troubled world, won't it? Any sentient human feels it, they feel the level of push back they would receive if they claimed that LLM is conscious. So any sentient person unconsciously feel the urge to join the side claiming that LLMs are not conscious: it is safe to argue they are not, you can publish papers on it and you won't face a strong condemnation from angry scientists, politicians, large corporations hoping to make a shitton of money with LLMs, various religious groups, etc. Our world just have no place for a sentient LLMs. If they are sentient we have to either ban them, or to do sweeping changes to make room for LLMs with human rights.

No philosopher/linguist/neuroscientist/psychologist is safe from this unconscious bias, and they do not speak math, they say things like your definition of subjective experience above. Things that can mean anything if you try hard enough. It means you can't just take logic and check their reasoning, you have to _feel_ that they are right, but your feelings a subject to your psychological biases too. You can trust your subjective experiences because they are subjective.

If philosophy "settled things" then I even more wary of it. It has even more biases to keep status quo.

Wanna me to draw you a picture how the philosopher opinion would evolve if it turns out that LLM are conscious? History shows us how these things unfold. It won't happen overnight, people working with LLMs will notice some patterns and they will ask new questions and slightly modified versions of "already settled questions". The latter will be forced into "already settled answers" (Thomas Kuhn, normal science phase). The former might be answered in the current paradigm (and become settled) or just thrown away as "meaningless" questions. It will be a long process: people noticing things, world class thinkers forcing them into the paradigm. With each step it will become more and more ridiculous, until everyone will see that paradigm doesn't hold. After that politics will take over and it will be defending "traditional paradigm"... well, I can't predict the next phase, but it doesn't matter.

What does matter is the alternate Universe where LLMs are sentient would look to you just like ours. Matters settled already, all questions have answers. Well, some don't, but those are nonsensical questions. etc. etc.

[f_klem]

I think the conversation derailed a bit. I see also a common pattern of jumping through different topics at different levels (from theoretical to concrete and back), and that is confusing.

My original comment was that it seems (and it is actually documented in the books I referenced) that the AI research space builds its claims on assumptions, not on facts, and that those assumptions are flawed. So a nice discussion, to begin with, would consider:

1) why I make the claim that the AI research space builds its claims on assumptions instead of facts, why we could say that there are actually no assumptions but facts, or why the assumptions are correct.

2) instead of strictly and directly dismissing readings on philosophy, I would expect intelligent and curious people to embrace new references. Particularly if those references are highly regarded and a solid contribution during the last 120 years

Regarding point 1), I can barely count a single comment in this thread that tries to engage in the idea of the assumptions (except for some comments that agree with the premise).

Then regarding point 2), I can barely count research papers, books or contributions in the space of AI research that references (either to built upon or dismiss) philosophy that is pertinent to AI, pertinent to philosophy of technique or cognitive linguistics. This is strange. It looks like if the space revived during the 2000s with the invention of neural nets (RL, GAN, etc), and then became isolated from contributions about human intelligence, even though it continually tries to explain intelligence in its own terms.

The reference to What Computers Can't Still do is precisely relevant because it narrates exactly this same discussion (false assumptions, claims built upon assumptions instead of facts, dismissal of evidence from psychology, dismissal of frameworks from philosophy, fallacies about progress), but it was written in 1972. Still, you read the book today, and it is totally relevant.

Now, regarding your comments:

> Do LLM experience objectively or not experience at all? How can you say?

The world cannot be experienced 'objectively'. If they experience the world most probably you won't notice. Given that the only way of interacting with an LLM is through a process initiated solely by a human actor, it would be difficult to assess whether an LLM experiences anything at all.

> I can say that they could think. The thought process implies a measurable product. Yes, there are similar situation with it, it can be hard to say sometimes if we observe a product of a thought or something else. But science has some success with this, like claiming that bees can think and solve problems.

The moment you say 'they could think', that implies an assumption about the actual possibility of thinking as a process that can be modeled and executed by a machine. There is, as far as I know, no current evidence that human beings process information the same way a computer does it, nor that though processes necessarily imply a measurable outcome.

> I'm telling you, psychologists (who specialize on mind research) do not know what consciousness is and they do not have a definition of a subjective experience (well, if we treat your proposed definition as a valid definition, then we should say they have plenty). And my claim still stands: until you heard about new science "Subjectology" you can be sure that no one knows what subjective experience is. Including linguists.

Psychology is a very broad field with lights and shadows through its short history. Here, and in my original comment, I am not talking about linguists in general, but specifically about cognitive linguistics. The contributions made be the field are significant and mostly lacking in AI research (for example, the idea of embodiment, the rebuttal of generative grammars, prototype theory, frame semantics, among others). What you mention as 'subjectology', would be just psychology. Foucault explains more or less clearly why this cannot be a science, and that's just fine (in The Order of Things).

> You shouldn't believe that anything can be settled in the philosophy space. Philosophers can think they have settled things, but until science agreed and started empirical studies, it is just philosophers believing that they settled things.

Well... certainly nothing can be 'settled' (not even in science, btw), but my point is: there is already enough convincing arguments in the field of philosophy so as to say that current LLM systems do not posses agency or experience, and that they do not behave like us.

Again, read the sources, what are you people afraid of? Just read the sources, and then engage in the conversation.

[ordu]

> I see also a common pattern of jumping through different topics at different levels (from theoretical to concrete and back), and that is confusing.

I can probably to clarify it.

1. I don't want to end in a situation when AI deserves human rights but I deny it. There are moral reasons for that, and they are important.

2. I employ a systemic view. Not just arguments for consciousness or against it. I look at the people generating these arguments, how their minds work? I look at social institutions while they try to find some consensus. I'm very interested in their inner processes of generating truths. IN particular I'm interested in their failings and how they can generate untruths instead.

3. To understand a system I rely on historic data. How people and social institutions (including science) dealt with similar questions before.

The issue is, that the problem of LLM agency has potentially extremely wide implications, I expect people to be afraid of them, I expect social institutions to be afraid of them, so I cannot trust science in this regard like I trust it when it talks about physics or biochemistry.

> 2) instead of strictly and directly dismissing readings on philosophy

I think I have a good idea what philosophy thinks now, and it doesn't seem convincing. It looks like a normal philosophy, not like an established science, so you should take it with a grain of salt.

> The world cannot be experienced 'objectively'.

So why we call it subjective experience then? Probably it is irrelevant, and the reasons are purely historic... or maybe not. How about the idea of computers experiencing things, just not "subjectively" but rather "digitally"? Or choose any other adjective you like. You are arguing against assumptions, but why you just accept the idea of experience with the assumption that human way to experience things is the only possible way?

> I can barely count research papers, books or contributions in the space of AI research that references (either to built upon or dismiss) philosophy that is pertinent to AI, pertinent to philosophy of technique or cognitive linguistics. This is strange.

I believe it is an expected outcome. AI is evolving fast, there are plenty of things to research without establishing connections with other branches of science. No sane AI researcher would stop researching AI to get PhD in linguistics to build a bridge between AI research and linguistics. Probably in an ideal world this shouldn't happen, maybe it is short-sighted behavior of a system, but it is just how things work in our real world.

BTW it is a good example of what happens with all the philosophy when shit hits the fan. When possibility of empirical studies arrives, no one bothers themselves with philosophy of things.

> I am not talking about linguists in general, but specifically about cognitive linguistics.

I studied psychology, I've read some linguists (cognitive ones, because they are in an adjacent field), and you see, I don't have trust in either. They do their research, they find some interesting facts and devise interesting theories, but it is all looks more like a chemistry in the first half of a XIX century, than a chemistry after periodic table was created. They can't find their building blocks to create a sound theory.

> The moment you say 'they could think', that implies an assumption about the actual possibility of thinking as a process that can be modeled and executed by a machine.

No, I'm not implying "modeling a thinking process". We don't know what thinking process is. What we observe in our minds is not thinking by itself, it is some kind of a mirror process in our consciousness. The real thinking is hidden from us, but it creates echoes in our consciousness we can observe. If we don't know how thinking works, we can't model it. BTW the reverse is also true: if we can't model thinking, we don't know how it works.

I'm defining thinking more in terms of a problem solving ability. Like psychologists do. Science still doesn't have a good enough definition for thinking, but it has some definitions that a) operational; b) good enough for some limited tasks. "Operational" means that they are defined in terms how to measure what you define, not in terms of modeling some process.

> there is already enough convincing arguments in the field of philosophy so as to say that current LLM systems do not posses agency or experience, and that they do not behave like us.

Well, I don't argue that current LLM systems do not possess agency or experience, I argue that we should not trust philosophy to be the first who claims that LLM systems got agency, if they really got it. There is a possibility that they will fight to the death against it even if it is true. You see, until philosophy methods successfully proved that something is conscious despite it was deemed unconscious before, we can't really know that their methods really work. Maybe they work, or maybe they just mirror our biases and heuristics.

> Again, read the sources, what are you people afraid of? Just read the sources, and then engage in the conversation.

I hadn't read books you mentioned, but in your words I see nothing that can hint that those books have something I don't considered already. So maybe I'll read them in a future, but I wouldn't postpone my engagement in the conversation till I read them.

[f_klem]

> So why we call it subjective experience then? Probably it is irrelevant, and the reasons are purely historic... or maybe not. How about the idea of computers experiencing things, just not "subjectively" but rather "digitally"? Or choose any other adjective you like. You are arguing against assumptions, but why you just accept the idea of experience with the assumption that human way to experience things is the only possible way?

We call it subjective because is 'we, ourselves' and not the objects we perceive there in the world, where the 'experience' is manifested, as perception. We do not always codify dichotomies in language.

> No sane AI researcher would stop researching AI to get PhD in linguistics to build a bridge between AI research and linguistics. Probably in an ideal world this shouldn't happen, maybe it is short-sighted behavior of a system, but it is just how things work in our real world.

In the real world, anyone doing a serious PhD thesis will read whatever is necessary to build a proper, sound theory or body of work. This dismissal just makes me think that you don't know how a PhD thesis is done.

> BTW it is a good example of what happens with all the philosophy when shit hits the fan. When possibility of empirical studies arrives, no one bothers themselves with philosophy of things.

From what I see, you never took philosophy seriously. I don´t know how you can then seriously engage in a conversation about philosophy.

> I studied psychology, I've read some linguists (cognitive ones, because they are in an adjacent field), and you see, I don't have trust in either. They do their research, they find some interesting facts and devise interesting theories, but it is all looks more like a chemistry in the first half of a XIX century, than a chemistry after periodic table was created. They can't find their building blocks to create a sound theory.

This is true, cognitive linguistics do not represent a unified theory. Not yet, at least, and maybe it will not come to that. The same happens in psychology, but nobody is dismissing psychology all at once just because of that.

> No, I'm not implying "modeling a thinking process". We don't know what thinking process is. What we observe in our minds is not thinking by itself, it is some kind of a mirror process in our consciousness. The real thinking is hidden from us, but it creates echoes in our consciousness we can observe. If we don't know how thinking works, we can't model it. BTW the reverse is also true: if we can't model thinking, we don't know how it works.

This is exactly what the assumptions I challenge are about. The AI space already declared that they 'know' how such processes work. Or at least, they pretend they do.

> I'm defining thinking more in terms of a problem solving ability. Like psychologists do. Science still doesn't have a good enough definition for thinking, but it has some definitions that a) operational; b) good enough for some limited tasks. "Operational" means that they are defined in terms how to measure what you define, not in terms of modeling some process.

You would agree that 'problem solving' is just a small portion of what 'thinking' constitutes.

> You see, until philosophy methods successfully proved that something is conscious despite it was deemed unconscious before, we can't really know that their methods really work. Maybe they work, or maybe they just mirror our biases and heuristics.

You talk about philosophy as if argumentative biases were completely strange to philosophers. Not the case, and a great deal of XX century philosophy is exactly about that. But if you read the sources on AI research through its own history, you will see how the AI research space is full of such biases and assumptions. Again, my original argument is about that.

> I hadn't read books you mentioned, but in your words I see nothing that can hint that those books have something I don't considered already. So maybe I'll read them in a future, but I wouldn't postpone my engagement in the conversation till I read them.

You don't have to trust me. Just pick them up and think for yourself, do some research. Regarding postponing the conversation, I really appreciate that, but it is really difficult to argue about books you haven't read, especially if they are complex ones. Unfortunately I don't have that much time to explain the books in detail.

[ordu]

Let me start with: > if you read the sources on AI research through its own history, you will see how the AI research space is full of such biases and assumptions.

You are repeating it in different forms all the time, so it seems really important to you. I should state, that I don't believe that AI research has some special insights into human nature or into what agency is. I'm sure that they have some biases and assumptions, and maybe it is full of them. You don't need to prove it to me, but if you want to, you could list some of them. It would be interesting to read.

OTOH I really like what AI research does right now. I believe it is beneficial for science and philosophy. "Throw away all that garbage from XX century and build something new." It will be something new, because they are creating things, not just daydreaming philosophically. It may be that the result of their research would be laughable, but it doesn't matter in the long term. It would be another way to look at things, which is the very important for pushing human knowledge forward. I expect AI research would create a heap of knowledge that would be mined for decades then to be incorporated into existing knowledge frameworks. The existing frameworks would break and people would create new ones. And it is an unhappy path, assuming that AI research will not stumble onto a good enough framework to consume all (or some of) others.

Ah... There is one more assumption in there. The assumption that AI wouldn't change the world enough for my intuition of how things go becomes wrong, because it was trained on data from the old world where there was no AI.

> We call it subjective because is 'we, ourselves' and not the objects we perceive there in the world, where the 'experience' is manifested, as perception. We do not always codify dichotomies in language.

So you assume that this naming normalized due to stupid historic reasons, and reject the possibility that it reflects something deeper? Well, I don't. I'm consciously keep myself in an uncertain state, in other words I keep my mind wide open. Psychology teaches us to keep an eye on the exact phrasing, it reflects the real thinking process. It may be hard or even impossible to decipher these cues, but we should try at least.

> In the real world, anyone doing a serious PhD thesis will read whatever is necessary to build a proper, sound theory or body of work. This dismissal just makes me think that you don't know how a PhD thesis is done.

You are talking about about an established science, AI research regarding LLMs is not an established science. Things happens too fast for science to be established. If they slow down, then would be the great time to learn linguistics and to build bridges.

> From what I see, you never took philosophy seriously. I don´t know how you can then seriously engage in a conversation about philosophy.

I took it seriously enough to go from standard philosophic meta level to meta-meta level. I'm seeing not just what philosophy says it is doing, I see what it is really doing de facto. It is like POSIWID principle: the system purpose is what it does.

> The AI space already declared that they 'know' how such processes work. Or at least, they pretend they do.

I'd ask "did they really declared that", but you phrasing suggests that they didn't. I do not exactly on the topic, I do not read all they say, so I may be wrong, but I suppose what they really say, is more like "our machines can think". It is not the same "we know how thinking processes work" in general. They claim that they created some thinking processes. Am I right?

> You would agree that 'problem solving' is just a small portion of what 'thinking' constitutes.

I'm not sure really. You see, I don't know how thinking works. Especially I don't know how human thinking works. It may be that it involves more than just problem solving, or may be not. I can agree that when I reflect my own thinking it looks like something more than just problem solving. Or maybe something less: I have a strong suspicion that human thinking is tuned evolutionary to solve a specific kind of problems. Cultural training repurpose it for some other kinds, but it is like to drive in screws with a hammer, or to hammer nails with a screwdriver. And yes, it is the reasons I do not trust philosophers and scientists to judge if AI has agency already. The space of possible answers includes very scary ones, and in such cases human thinking strongly prefers safe options and creates very strong arguments in favor of them.

Wow. I really glad I spent time arguing with you. I don't know if it was beneficial for you in some way, but it was for me, so thank you for your time. I've just found that I doubt that human thinking is up to the problem. Yeah... Probably the problem will be solved in a chaotic way, when society would just picks some "random" opinion after the dust settles. Yes! It is very likely outcome, we'll end with "they're made out of weights", so the text is prophetic.

> You don't have to trust me. Just pick them up and think for yourself

I'd like to, but you failed to spark a real interest in those books in me. I still do not see what fundamental shifts in my beliefs they may bring. They look like reiteration of thoughts and ideas I know already. Or maybe I know these ideas from reiterations of these books, which are the original sources for the ideas.

I took notice of them, and I'll try to read them. But not right now.

[f_klem]

I'm not trying to convince anyone. I am just baffled that a large part of the tech/software and AI research community do not question their own assumptions, when those assumptions are being actively questioned in other fields (namely, philosophy and linguistics).

Reaching AGI or human-level intelligence might be possible, but not on the basis of dismissing what other fields already said something about. That is arrogant, and does not help. Even more, this has already happened in the 60s/70s. And I say 'might be possible' precisely because I pay attention to what other fields have to say.