[JulianChastain]

A charitable interpretation of nico is that he was saying a well-trained NN is itself a model of the world. If it can tell you what a system will do given some inputs, then it functions as a model. While internally it isn't creating a model that we could understand, it does "model the world" in the sense that we can treat it as a model

[PaulDavisThe1st]

Consider for a moment replacing the NN with another person, who forms a model of the world that is very useful for prediction.

Now our lead experimenter asks this person "what will happen if the global average temperature increases by N degreesC?" and they get an answer.

Can we way that the lead experimenter has built a model? They have not, certainly not in the sense that they have any access to it. The person who replaced the NN may have (and indeed, probably has) built some sort of model, but that's a very different claim.

Explainability in NN/ML systems is a hot topic, and many people (not all!) would say that if the NN/ML system cannot explain why adjusting parameter X will cause changes in parameters A, M and T, then you have no access to anything that merits being called a model.

A consequence of this is that if the person who replaced the NN can explain themselves (e.g. answer the X -> A,M,T coupling), then even the experimenter can probably be said to "have a model". But if all that can be said is "I don't know and/or I can't explain, you just need to trust me that this coupling is real", then the claim that a model has been built is on unstable ground.

[onecommentman]

A truism in the computer modeling communities of the 1970s and 1980s was “the product of a modeling exercise is not the computer model, but the modeler”.

The insight gained by rigorously modeling a system in computer code produces a person (the modeler) who can provide valuable insight when asked questions about the system. In policy analysis, the modeler’s insight can often provide quick and dirty and auditable (and often correct) analyses/answers about the modeled system without ever running the developed formal computer model. The exercise of the formal development of a computer model credentials the modeler as having gained a level of rigorous systems-level expertise. And the scope and detail of that modeler knowledge is certified in the depth and breadth of the computer model itself (and the currency and accuracy of the input data sets).

Nice to have such an human analyst around when important policy decisions need to be made, since such policy decisions should be made and implemented by humans who can explain the confidence that exists regarding the knowledge that supports the given decision. The decision makers can then point to the analysts for the estimate of the degree of confidence that can be ascribed to the policy analysis that supports the decision. That’s how it’s supposed to work, and that philosophy is formalized in existing decision processes for complex technical systems such as transportation, telecommunications, power, military systems, etc.. You know, the important stuff….

[PaulDavisThe1st]

All of which is precisely my point ... the NN/ML pathway does not lead to a modeller who can provide valuable insight when asked about the system.

[nico]

This is an awesome concept. Thank you

[nico]

> Now our lead experimenter asks this person "what will happen if the global average temperature increases by N degreesC?" and they get an answer

What symbols or language did the lead experimenter use to ask the person the question? And what does degrees, temperature and global mean?

All of those things require models to be communicated between the components of your system

Any symbolic communication is necessarily a model of what it is trying to represent

Of course, if there isn’t someone to interpret it, it’s just symbols. But to interpret a meaning behind symbols, then it implies the symbols represent a model of the meanings that are being communicated

[nico]

That’s an excellent interpretation. Thank you