[mitthrowaway2]

Spock is fictional but his writers weren't! They're the ones whose processes get simulated, which is why it would output technobabble on such a prompt instead of actually-good ideas that come from a Vulcan from the future. It can also simulate the style of Rudyard Kipling or whoever else you choose who is non-fictional and with a distinct enough style.

And, I'd argue, so can many of us humans! After reading a Jane Austen novel, it can take a conscious effort not to write in the style of Austen. ChatGPT manages it better than I do. I don't think I know her well enough to get into her brain, but it seems like there's something like a transfer function called STYLE between "the message Jane Austen wants to write" and "the words Jane Austen chooses to write".

                        _____ 
  intended message --> |STYLE| --> selected words
                       |_____|

This STYLE transformation is clearly modular enough that it can be easily swapped out for someone else's, and sufficiently non-mysterious that you, I, and ChatGPT can all recognize and pretty accurately emulate it.

I don't think ChatGPT can simulate Jane Austen well enough to tell us her opinions about her childhood or any other message that she might have generated, but it seems to be able to replicate very closely the steps that Jane Austen's own mind herself was following as part of that STYLE.

ChatGPT does seem to go even further than this, because it also has some understanding of where different sorts of characters would steer the message of a conversation. But while it's believable, it's hard to say how accurate that is to what any particular real person would say.

[galaxyLogic]

You can IMITATE the outputs of an author, but that is not the same thing as SIMULATING said author. When talking about LLM AI it is often implied that LLMs are "intelligent", that they are like (the truly) intelligent humans because they are "simulating" such intelligence.

But IMITATING the output of something is not the same as SIMULATING the process that produces that output.

Taking a photograph or creating a movie imitates the reality around us. It does not simulate the processes that produce the look and feel of our reality.

[mitthrowaway2]

There is a difference. But the more alike two processes are in their input-output behavior, the more likely it is those processes are alike on the inside as well. If process B matches the input-output behavior of process A, it's imitating process A. If it is following an equivalent sequence of steps in order to generate those outputs, it's simulating process A.

The harder it is to discriminate between A and B on a long series of diverse inputs, the more likely it is that A and B are internally equivalent, not just externally similar. The reason is that there's no better fit than B = A.

I'm increasingly doubting whether my own brain might not, internally, use something that is architecturally similar to an LLM in order to compose comments like the one I'm writing now.

[galaxyLogic]

I can see the appeal of that kind of thinking. Babies learn words by repeating them without knowing what they mean. They gradually learn the meaning of words by trying to use them and getting feedback. But LLMs are not trying to "use" their language for any particular purpose. They just idly chat on, like a machine :-)

It is possible to repeat words and sentences without having any idea of what they mean. I think the LLMs are currently at that stage.

[mitthrowaway2]

A great deal of feedback happens during training.