[f_klem]

The issue of consciousness appears when you think of the world in a mechanistic way: since all there is are laws of physics and materiality, then how could we explain our though processes and our perceptual experience? If the world itself (in a general, existential way) is only made of laws of physics and matter, the consciousness needs to be an emergent characteristic of physical systems, and needs to follow the laws of physics. Now at this stage, you are already in trouble and you need to explain what consciousness is and how it manifestates. And that's the moment where things like the computational theory of mind appears.

But you need to step back in order to detect the fallacy, one of which is: the brain/mind processes information like a computer, then we could build better computers that can think. This fallacy is assumed in the question 'can a machine think?'. There's another fallacy, which the author call the first step fallacy, which is common nowadays: we solved the language problem, then machines will be able to think in the near future.

So it is not about solving the consciousness problem, it is about not claiming things based on assumptions that can be easily challenged.

[LudwigNagasena]

Of course any research programme requires some assumptions. But I don’t see any reason to call it a fallacy. Saying that something may be “challenged” or is problematic is just weasel wording.

Either there are some serious issues that makes such theories ”flawed in the sense that they cannot account for subjective experience and agency, amongst other things”, or they are just normal theories.

[f_klem]

True, any research programme requires assumptions. The problem lies when those assumptions are either false or theoretical (unproved), and the community derives facts or claims from them.

Behind the actual AI programme operate the following assumptions (at least):

1. A biological assumption, that states that the brain works similar to a digital computer. The reality is that we do not know.

2. An epistemological assumption, that states that we know how our brain works (or an even worse assumption, that states that we don't even need to know how it works, it is sufficient to replicate its observed behavior). This is rather simplified, the assumption in reality being (as stated by Dreyfus) that we think all intelligent behavior can be formalized as heuristic rules (Dreyfus' critique is based on GOFAI, since the book is pre-GAN/RL AI systems). But the assumption still applies: we think all intelligent behavior can be sampled, captured and formalized in (albeit complex) statistical systems.

Dreyfus describes 4 or 5 in total, one of them is the psychological assumption, which states that the mind itself can be described as a digital computer (I think it might be outdated, since the actual debate is if something we could call 'mind' exists at all).

There is also a fallacy called first-step fallacy, which states that if the first step towards intelligence is met, then the rest of the steps are of similar nature (technical).

[LudwigNagasena]

You say that "the community" derives facts or claims from unproved assumptions, yet at the same time you say that you "strongly tend to disagree" with those theories and that the theories are "flawed in the sense that they cannot account for subjective experience and agency, amongst other things" merely on account that they are neither confirmed nor unconfirmed. I am confused about your stance. You allow yourself to have strong opinions about something unknown yet criticize other people for the same.

I think it is absolutely normal that the core of a theory is based on not directly testable assumptions. And it's normal that people push it forward if it bears fruits, that's not a fallacy in any way, that's normal inquiry that may or may not lead to successful results.

[f_klem]

> You say that "the community" derives facts or claims from unproved assumptions, yet at the same time you say that you "strongly tend to disagree" with those theories and that the theories are "flawed in the sense that they cannot account for subjective experience and agency, amongst other things" merely on account that they are neither confirmed nor unconfirmed. I am confused about your stance. You allow yourself to have strong opinions about something unknown yet criticize other people for the same.

The assumptions I refer to are not only unproved, there is also increasing evidence that they are false. I do not criticize based on the assumption that there is subjective experience, but on the well developed idea that there must be something like 'subjective experience'. Here we enter the realm of philosophy, which by the way, is what science encounters when it runs out of answers. And this was precisely my point: AI research is based on assumptions that _need support or help_ from philosophy, not only neuroscience. But what is at stake here is the prevailing neurocentrism and scientificism characteristic of our era.

> I think it is absolutely normal that the core of a theory is based on not directly testable assumptions. And it's normal that people push it forward if it bears fruits, that's not a fallacy in any way, that's normal inquiry that may or may not lead to successful results.

That is correct and it is precisely why they are called 'theories': because the evidence points towards a specific direction but there is not yet enough evidence to call it a law.

Yet different theories, based on different assumptions, demand that those assumptions be tested at the fundamental level: logical, epistemological, philosophical, etc.

Regarding the theory that current LLM research could lead to human level intelligence, many people have the opinion that it can be discarded on fundamental grounds. Why? Because the assumptions that this theory stands on are flawed.

An issue I repeatedly see in the community (about which Dreyfus already wrote in his 1972 book, confirmed in his 1992 book, and we still see today) is that challenging the fundamental flaws in which current AI research is based on immediately sparks outrage in the AI community, as if people challenging those assumptions are against AI or AI research at all. I think that is a really silly, childish and not very humble position, and ultimately slows down research.

[LudwigNagasena]

Now you again say that there is increasing evidence that the assumptions are wrong and that the foundation is flawed, but when you get to specifics you merely claim that something is unknown or unconfirmed.

[f_klem]

The books that I referenced at my first comment already contain an extensive explanation of why those assumptions are flawed, false, or ungrounded.

I will not paste here parts of books.

Nonetheless, I've been compiling references on different AI research assumptions and problems. I'll paste them here later on.