[microtonal]

Suppose that we used embeddings as the input of the model rather than piece identifiers plus an embedding lookup table. This is possible with every transformer model and some libraries provide an API to do this. Moreover, we convert the parameters and ops to use arbitrary precision types. Then the network cannot be represented as a lookup table. Given that there is an infinite number of inputs, there is also an infinite number of outputs. But the arbitrary-precision network does not operate fundamentally different from the original network. It has the same parameters, ops, etc., yet you cannot store it as a (finite) lookup table.

[clooper]

Even if you increase the precision I can still generate a table T(P) for each fixed precision P. So the table is parametrized by P but it's still a table. The entire table T = colim T(P) is the colimit over all precision values but for every finite precision it is still a table.

[microtonal]

I did not say fixed precision. I said arbitrary precision, so P is infinite.

The only counter-argument is that even arbitrary precision is fixed-precision because computer memory is finite. But that's kind of a silly argument, because then you are arguing that computers can never reason, because they have finite memory, and moreover humans cannot reason either, because there is a finite number of brain cells.

[clooper]

P obviously can't be infinite, even in theory, if you want the computation to terminate.

[microtonal]

Right, but then as others said, then you are also arguing that humans cannot reason, since the universe is a system with a finite number of particles. Or if we exclude external factors, because humans have a finite number of brain cells.

In the end it all depends on what your definition of reasoning is, which you did not provide.

[clooper]

The bit precision of computation is always finite for halting computations and any finite computation can be turned into a lookup table which does no thinking or reasoning other than comparing two numbers and then extracting the value corresponding to the input key.

My argument carries through for any piece of software so if you think software can think and reason then you can remain unconvinced by my argument.

In any case, I have to drop out of this thread.