[strogonoff]

All communication is inherently lossy, and text is extremely so. Knowledge, insight, etc., is never captured in its entirety in communication. Indeed, there is no direct contact between human minds, not in the models we currently have.

Communication builds on simplified shared maps over ineffable territory of human experience. It always presents a particular model—a necessarily wrong one (as all models are), good for one purpose but neutral or harmful for another.

However, models and maps is not the only way in which humans attend to reality. Even though it is compelling to talk as if it was the only way—talking is communication, and naturally it likes communicable things—we also have the impossible to convey direct experience. Over the past thousand or two years, as humanity becomes more of an interconnected anthill, this experiencing arguably increasingly takes a backseat to map-driven communication-driven frame of attention, but it still exists and is part of what makes us human.

LLMs, as correctly noted, build only on our communication. What I don’t think is noted, is that this means they build on those (inevitably faulty) models and maps; LLMs fundamentally have no access to the experiencing aspect, and the territory-to-map workflow is inaccessible to them. What happens when wrong maps overstay their welcome?