[amatecha]

My layman thought about that is that, with consciousness, the medium IS the consciousness -- the actual intelligence is in the tangible material of the "circuitry" of the brain. What we call consciousness is an emergent property of an unbelievably complex organ (that we will probably never fully understand or be able to precisely model). Any models that attempt to replicate those phenomena will be of lower fidelity and/or breadth than "true intelligence" (though intelligence is quite variable, of course)... But you get what I mean, right? Our software/hardware models will always be orders of magnitude less precise or exhaustive than what already happens organically in the brain of an intelligent life form. I don't think AGI is strictly impossible, but it will always be a subset or abstraction of "real"/natural intelligence.

[walkabout]

I think it's also the case that you can't replicate something actually happening, by describing it.

Baseball stats aren't a baseball game. Baseball stats so detailed that they describe the position of every subatomic particle to the Planck scale during every instant of the game to arbitrarily complete resolution still aren't a baseball game. They're, like, a whole bunch of graphite smeared on a whole bunch of paper or whatever. A computer reading that recording and rendering it on a screen... still isn't a baseball game, at all, not even a little. Rendering it on a holodeck? Nope, 0% closer to actually being the thing, though it's representing it in ways we might find more useful or appealing.

We might find a way to create a conscious computer! Or at least an intelligent one! But I just don't see it in LLMs. We've made a very fancy baseball-stats presenter. That's not nothing, but it's not intelligence, and certainly not consciousness. It's not doing those things, at all.

[tavavex]

I think you're tossing around words like "always" or "never" too lightly, with no justification behind them. Why do you think that no matter how much effort is spent, fully understanding the human brain will always be impossible? Always is a really long time. As long as we keep doing research to increasingly precisely model the universe around us, I don't see what would stop this from happening, even if it takes many centuries or millennia. Most people who argue this justify their point by asserting that there is some unprovable quality of the human brain which can't be modeled at all and can only be created in one way - which both lacks substance and seems arbitrary, since I don't think that this relationship provably exists for anything else that we do know about. It seems like a way to justify that humans and only humans are special.

[kraquepype]

This is how I (also as a layman) look at it as well.

AI right now is limited to trained neural networks, and while they function sort of like a brain, there is no neurogenesis. The trained neural network cannot grow, cannot expand on it's own, and is restrained by the silicon it is running on.

I believe that true AGI will require hardware and models that are able to learn, grow and evolve organically. The next step required for that in my opinion is biocomputing.