| > But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens. There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning. I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task. Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes. [1] https://arxiv.org/pdf/2301.05217.pdf |
There's an interesting theory that I've never been able to identify the origins of, that humans like to think that we created technology from needs based on our models of the world, whereas we ignore the effects of daily technology on our thinking frameworks. The argument essentially states that we never "discovered" the circulatory system of the heart or "discovered" it works like a pump and valves, rather instead right around the same time this theoretical work was being investigated on the heart is when the industrial revolution was in full swing. Thus, we modelled the heart as pumps and valves because that's the technology we were surrounded by. The heart isn't somehow inherently a "pump" and we "discovered" that, we just started using the pump metaphor because it seemed to help do other things. I'm sure though that the metaphor has it limits.
Typically, the narrative around the invention of machine learning models is that we started coding computers to be more like recently "discovered" models of the brain.
Under this theory its the opposite. Right as cognitive sciences started developing as a novel field of research is when we developed computers. So, in classic form we decomposed the brain in atomic fashion, and the 'atoms' of measurement we chose to use ended up being bits and bytes.
Distinguishing between inventing "novel" things and gobbledygook is completely subjective and based on the viewer's own models. It's proving these abilities after the fact, not before it. Thousand monkeys on typewriters etc.
This measurement of "accuracy" is completely forgetting everything that Kuhn discovered about scientific knowledge. If you've got a community of like 3 people who research some incredibly esoteric scientific field, only those 3 people could ever accurately judge the full extent of their domain. A model could generate a series of tokens that for the rest of the world is gobbledygook, but to these 3 scientists it makes perfect sense. This doesn't really endeavour me to believe that there's anything "novel" about what AI "predicts". It just throws out enough combinations and we conveniently ignore the huge gaps when its wrong, but then jump up and down excitedly when it's "right" (as if its discovered some universal material objective truth).