[skyhook_mockups]

Thanks for your answer, its given me something to think about.

A few things immediately come to mind though:

A sentence is just letters, but that doesn't mean that thorough examination of each letter provides you with the knowledge necessary to understand the sentence

I see what you're saying here, but consider this example:

Give someone a box of clock components and a clock from which to draw inspiration, and without any understanding of how a clock works or how the cogs and springs are manufactured etc, they will, given enough perseverance build a working clock.

This simple analogy illustrates that understanding exactly the functioning of each sub-component of a system is not necessary to be able to exploit its usefulness.

[andolanra]

I was primarily making an argument the other way around: I said that understanding the subcomponents (i.e. the letters) did not guarantee understanding of the whole (i.e. the sentence), while you're giving an example in which someone builds a whole, presumably by understanding and replicating the connections of major components, but does not understand the parts. More analogous to my point would be attempting to understand the workings of a clock by examining each gear.

Also, I don't know that your thought experiment holds water. I can certainly conceive of a universe in which a person never makes the logical leap from holding a clock and parts—or even having a thorough understanding of the workings of the subcomponents of a clock—to building their own. The pre-Columbian New World civilizations, for example, had all the resources to build wheeled vehicles, and certainly understood the principle behind them enough to build wheeled toys or use rolling logs to transport large objects, and the Inca Empire even had a sprawling complex of roads—but never in their long history did the notion of a wheeled cart occur to any of them. Which is to say: the search space of ideas is vast, and one can't reasonably be expected to exhaust them all even with help.

[skyhook_mockups]

That's not really the point I was making. My point is that you don't have to understand the workings of a clock (or its subcomponents) to be able to build one given the subcomponents and an example from which to work. Simply mimicking exactly what you observe will produce the desired outcome.

There are lots of examples where the underlying workings are not understood, and yet useful work is done. Look at medicine for example. For thousands of years people knew that if you mix herb A and B and boil them for time X you get something that fights infection or helps with headache. The underlying biochemistry doesn't need to be known or understood to be able to follow the steps to get the desired outcome.

The same might hold true for the brain. If we can catalogue all the connections and information pathways (chemical, electrical), we may not need to be able to explain how every combination of subcomponents interact, but, we may know that a particular arrangement gives the outcome we're after.

PS: "the search space of ideas is vast, and one can't reasonably be expected to exhaust them all even with help."

That's true but in this example we're not searching for anything. Over millions of years one solution to intelligence, from an infinity of other possibilities has already evolved. Each of us already has a working version of what we want to replicate in our own skull. Now we need to tease out all of the cogs and springs and how they are arranged, arrange them in the same ways, and unless there truly is a metaphysical component we will have something indistinguishable from human intelligence/consciousness.