[mitthrowaway2]

There is a difference. But the more alike two processes are in their input-output behavior, the more likely it is those processes are alike on the inside as well. If process B matches the input-output behavior of process A, it's imitating process A. If it is following an equivalent sequence of steps in order to generate those outputs, it's simulating process A.

The harder it is to discriminate between A and B on a long series of diverse inputs, the more likely it is that A and B are internally equivalent, not just externally similar. The reason is that there's no better fit than B = A.

I'm increasingly doubting whether my own brain might not, internally, use something that is architecturally similar to an LLM in order to compose comments like the one I'm writing now.

[galaxyLogic]

I can see the appeal of that kind of thinking. Babies learn words by repeating them without knowing what they mean. They gradually learn the meaning of words by trying to use them and getting feedback. But LLMs are not trying to "use" their language for any particular purpose. They just idly chat on, like a machine :-)

It is possible to repeat words and sentences without having any idea of what they mean. I think the LLMs are currently at that stage.

[mitthrowaway2]

A great deal of feedback happens during training.