[thornewolf]

> is qualitatively different from a robot

yes.

> To reduce a system to its inputs and outputs is fine if those are all that matter in a given context

we argue that this indeed is all that matters

> but in doing so you may fail to understand its internal mechanics

the internal mechanics are what we call "conscious" it is the grouping of internal mechanics into one unified concept, but we don't care exactly what they are.

> Those matter if you're trying to really understand the system, no?

since we cannot directly observe consciousness, we are forced to concede that we will never really "understand" it outside of observing its effects.

In the same way that a mechanical turk human and a robot can "play chess", a human and an LLM are "conscious". That is, consciousness is the ability to play chess, by some mechanism. The exact mechanism is irrelevant for the purposes of yes/no conscious.

We now enter a discussion on how much these two consciousnesses differ.

[darkwater]

> since we cannot directly observe consciousness, we are forced to concede that we will never really "understand" it outside of observing its effects.

Why? You are using a definitive term ("never") to something that we might achieve in a future. We might observe consciousness in a future. Who knows? Consciousness is a known unknown. We know there is something but we don't know how to observe it properly and how we could eventually copy it.

In the meanwhile, we are not copying consciousness, we have a shallow replication of its output. When cavemen replicated the fire that they observed as the output of a lightning, did they master electricity?

[TimTheTinker]

> since we cannot directly observe consciousness

But we do agree that it exists. Our direct experience tells us so.

> we are forced to concede that we will never really "understand" it outside of observing its effects.

Not necessarily. A gap in our ability to observe something does not imply that (a) we never will observe it or (b) what we don't know is not worth knowing.

Throughout history, persistent known-unknowns have pushed people to appeal directly to the supernatural, which short-circuits further discovery when they stop there. But the real fallacy is saying "we don't know, and it doesn't matter". That's a far more direct short-circuit to gaining knowledge. And in both cases, a lack of curiosity is an underlying problem.