[attemptone]

Not OP, but I see the also in problem with 'every possible entity'.

If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial.

Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it can generate every possible output in ASCII, most with very low probability but still.

After writing this, I'm not even sure what 'simulating' means in this context.

[mitthrowaway2]

I think "simulating" in this context means internally executing a process that is very similar to the process that generated the original material, as part of the prediction process. In general, that's the most compact way to predict and reproduce the original material.

For example, the string "1010101010"... could be the output of a function

  def generate_char_random(prev_string):
      x = random()
      if (x > 0.5):
         yield(1)
      else:
         yield(0)

It could also be the output of this function:

  def generate_char_alternating(prev_string):
      x = float(prev_string[-1])
      if (x < 0.5):
         yield(1)
      else:
         yield(0)

Even if it's not explicitly running those two functions, a model that is very good at predicting the next character of this input string might have, embedded within it, analogues of both of those two functions. The longer the output continues to follow the "101010" pattern, the higher confidence it should place on the _alternating version. On the other hand, if it encounters a "...110001..." sequence, it should switch to placing much more confidence on the _random version.

The LLM of course does not contain an infinite list of generative functions and weight their outputs. But to the extent that it works well and compactly approximates Bayesian reasoning, it should approximate a program that does.

[remexre]

"Every possible entity consistent with the distribution of input data it's been trained with," perhaps?

Simulating as in, having equivalent (or "similar enough") input-output behavior, I'd assume.

[galaxyLogic]

"Simulating" has a clear definition, but in this case what is it simulating? "Text-generating entities"? What are these text-generating entities it is (supposedly) simulating? Can you tell me where I can find one? Is it a person like me who writes this reply? So is it trying to simulate me personally?

Or are you thinking that it is simulating the aggregated behavior of all humans whose text-outputs are stored on the internet?

Are we saying it is simulating the combined input-output -behavior of all humans whose writings appear on the internet? But does such an "entity" exist and does it have behavior? I write this post and you answer. It is you who answers, not some mythical text-generator-entity that is responsible for all texts on the internet. There is no such entity is there?

It does not make sense to say that we are simulating the behavior of some non-existent entity. Non-existent entities do not have behavior, therefore we can not simulate them.

[mitthrowaway2]

There clearly exists a computable function that is a good enough approximation of "galaxyLogic's reply to remexre's comment" that it might be hard to for me tell whether the output was generated by the human brain or by an LLM. That function might indeed end up reproducing the same steps that your brain follows in constructing a reply.

(Just speaking hypothetically here).

While we understand LLMs, we don't understand the human brain, and in particular I don't think we've yet proven that human brains don't contain embedded routines that are similar to LLMs.

Someone with your particular writing style might be one, of several, simulations that are approximated within the LLM. Just like I can have it respond in the style of Spock from Star Trek.

[galaxyLogic]

I would say the LLM output may resemble the speech of the fictional character Spock. But it does not and can not simulate Spock, because Spock does not exist, never did. Spock is fictional.

To produce something that resembles the output of the fictional character Spock is straightforward, just take the texts that are parts of the fiction where fictional Spock speaks, and reassemble then using probabilities that can be calculated by statistically analyzing those texts. That is what LLMs are doing, right? And results can be quite surprising. I assume people were similarly impressed when they first saw movies.

But LLMs are not simulating anything, just like a movie or a photograph are not simulating anything, even though they may PROJECT the visual appearance of their subjects.

Are movies AI? I think it is clear to us they are not even though the characters on the screen seem to behave very intelligently. Movies are about representing and portraying the appearance of real or fictional events in the world. Similarly LLMs are about portraying texts on the internet. LLMs in my opinion are more like interactive movies than simulations of intelligence.

I do believe "true AI" will come eventually, and LLMs can give us an impression of what it might look like when it arrives, just like movies can give us an impression of Spock, who doesn't exist.

[mitthrowaway2]

Spock is fictional but his writers weren't! They're the ones whose processes get simulated, which is why it would output technobabble on such a prompt instead of actually-good ideas that come from a Vulcan from the future. It can also simulate the style of Rudyard Kipling or whoever else you choose who is non-fictional and with a distinct enough style.

And, I'd argue, so can many of us humans! After reading a Jane Austen novel, it can take a conscious effort not to write in the style of Austen. ChatGPT manages it better than I do. I don't think I know her well enough to get into her brain, but it seems like there's something like a transfer function called STYLE between "the message Jane Austen wants to write" and "the words Jane Austen chooses to write".

                        _____ 
  intended message --> |STYLE| --> selected words
                       |_____|

This STYLE transformation is clearly modular enough that it can be easily swapped out for someone else's, and sufficiently non-mysterious that you, I, and ChatGPT can all recognize and pretty accurately emulate it.

I don't think ChatGPT can simulate Jane Austen well enough to tell us her opinions about her childhood or any other message that she might have generated, but it seems to be able to replicate very closely the steps that Jane Austen's own mind herself was following as part of that STYLE.

ChatGPT does seem to go even further than this, because it also has some understanding of where different sorts of characters would steer the message of a conversation. But while it's believable, it's hard to say how accurate that is to what any particular real person would say.

[galaxyLogic]

You can IMITATE the outputs of an author, but that is not the same thing as SIMULATING said author. When talking about LLM AI it is often implied that LLMs are "intelligent", that they are like (the truly) intelligent humans because they are "simulating" such intelligence.

But IMITATING the output of something is not the same as SIMULATING the process that produces that output.

Taking a photograph or creating a movie imitates the reality around us. It does not simulate the processes that produce the look and feel of our reality.