[lxgr]

> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with

It’s less about being too tired and more about being realistic about the limits of understanding.

Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.

[narrator]

If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.

[noduerme]

When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.

Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.

People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.

Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

[TeMPOraL]

> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.

Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.

Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.

People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.

Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.

[nathan_compton]

I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.

I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.

> that we glean insights into nature of the simulated phenomena

That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.

[noduerme]

Great post. And that's exactly where I think we are with language models... we as a civilization are hypnotized and enchanted by the overfitting of models whose parameters are beyond our understanding, but whose mistakes we are more likely to forget than its accuracies, which again is a central human characteristic that explains our attraction to both psychics and slot machines.

Heck, it even explains my own attraction to overfit sports betting algorithms. No one is immune.

What's dangerous is when something like that replaces independent thought and becomes societally pervasive. That's an "oracle" the likes of which ancient civilizations warned that believing would lead to tragedy (or at a minimum, accidentally boning your own mother).

I'm an atheist, but raised Jewish. I read the Torah as a series of specific warnings and prohibitions against every type of shamanism, magic, witchcraft, prognostication, and deification of systems which predict (as well as systems which attempt to turn language into machinery, and worship the machine they've built ... see also, "Sound of Silence" by Paul Simon and "The Future" by Leonard Cohen, which both express this theme well). The framework requiring proof and disavowing illusion or the belief that all is illusory is notably different from a Buddhist perspective, for example.

We as a culture, right now, are not handling well the rise of a golden idol or an oracle in our midst. The right response is to try to trace the output back to ground truth and figure out why your model made a prediction... or else to build a model from ground truth and see how it performs against the oracle. We are doing neither. We're diving headlong into our own confirmation biases.

[Edit] I just wanted to add, because I got off track, that your conclusion about what's going on with human curiosity in cases where prediction is not the issue seems right to me. Barring some edge cases like predicting an eclipse and using it to slaughter your enemies, I think a lot of us do simply want to understand how things work, because figuring them out is enormously gratifying and is the work of lifetimes of incredible people who came before us. Using that knowledge or those techniques to predict things is technology, not science, and while I'm a fan of both, the former is only ever a practical test of the latter. Moreover, the sense of accomplishment of randomly walking your way to a profitable model is ephemeral and in a way earthbound, limited to the plane of one's own brief existence. Even if it were platonically perfect, a model is only saying how something behaves, not how it works. That's nothing compared to the joy of figuring out even the most trivial or axiomatic thing about how a cell or a compound or a physical structure or anything works, about how the universe actually works. And I think our better angels tell us to seek those answers, because our own life is fleeting, and predicting behavior is, like wealth, something you can't take with you. And not something you'll be remembered for anyway.

[noduerme]

I think we're talking about different kinds of models. I was referring to things like fluid dynamics equations that explain why gases and liquids move and how they act when changing states, as a basis for building weather models that predict how things will unfold in the future.

I'm also a fan of going the other direction: I've had a sideline working on code to evolve genetic algorithms for the past 20 years, and while the goal of that is to be predictive and profitable, it's often the underlying real-world dynamics my little mutants surface which are the most interesting and applicable in the long run. So I'm not saying there isn't a place for throwing everything at the wall until you see what sticks and then deriving a hypothesis from that (whether your interest is to predict the future, or merely academic, to explain the past). What I am saying is similar to you: We should not treat any model as an oracle. But I'm also saying that models can be built or they can be evolved, and if we only evolve them without understanding how they work, we are missing a crucial ingredient to knowing how well we should rank them. Overfitting and sample bias and data leakage are not problems when you want an equation to calculate airflow over a wing. If you began with an evolved equation which derived the results and didn't start from the base reality, you couldn't trust that equation to be airworthy even if it were right 99.99% of the time against the data it was trained on.

[antonvs]

> The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given.

This is just your own idiosyncratic and biased belief. You're not describing anything objective about LLMs, you're describing your personal attitude to them. This colors your understanding in a way that can't really be reasoned with until you let go of the artificial constraints you're imposing on your own understanding.

[lxgr]

> Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

If the LLM in their pocket has a more robust world model than they themselves and is e.g. able to refute their irrational convictions, it actually seems like a very good idea. (Big if, of course.)

[noduerme]

I actually wish I could upvote this more. It's a great point.

Yes, if the LLMs didn't amplify whatever people already thought and feed it back to them as sycophantic praise, and instead scolded them into realizing they were doing something dumb, then maybe we would be having a totally different conversation.

But then the conversation might be about an LLM scolding someone into committing suicide instead of helping them commit suicide.

Both might be just as bad.