[TZubiri]

"We're talking about specific outputs generated by the LLM, not the LLM itself. The training data consists of prior expressions of language which in turn may be influenced by human observations of reality, but the LLM is only ever making probabilistic inferences based on that second-order data"

You recognize that training data are influenced by human observations. And that LLM outputs are influenced by training data (and fine tuning). So it follows that LLM outputs are influenced by observations of the world. Why would the causality chain stop after 2 links?

https://chatgpt.com/share/67534483-8e6c-800f-9534-d764a90981...

You may call this a hallucination, but it is for sure based on observation. Otherwise the LLM wouldn't know the answer. It is undeniable that LLMs have empirical knowledge of the world through embedded human observation.