| Consider for a moment replacing the NN with another person, who forms a model of the world that is very useful for prediction. Now our lead experimenter asks this person "what will happen if the global average temperature increases by N degreesC?" and they get an answer. Can we way that the lead experimenter has built a model? They have not, certainly not in the sense that they have any access to it. The person who replaced the NN may have (and indeed, probably has) built some sort of model, but that's a very different claim. Explainability in NN/ML systems is a hot topic, and many people (not all!) would say that if the NN/ML system cannot explain why adjusting parameter X will cause changes in parameters A, M and T, then you have no access to anything that merits being called a model. A consequence of this is that if the person who replaced the NN can explain themselves (e.g. answer the X -> A,M,T coupling), then even the experimenter can probably be said to "have a model". But if all that can be said is "I don't know and/or I can't explain, you just need to trust me that this coupling is real", then the claim that a model has been built is on unstable ground. |
The insight gained by rigorously modeling a system in computer code produces a person (the modeler) who can provide valuable insight when asked questions about the system. In policy analysis, the modeler’s insight can often provide quick and dirty and auditable (and often correct) analyses/answers about the modeled system without ever running the developed formal computer model. The exercise of the formal development of a computer model credentials the modeler as having gained a level of rigorous systems-level expertise. And the scope and detail of that modeler knowledge is certified in the depth and breadth of the computer model itself (and the currency and accuracy of the input data sets).
Nice to have such an human analyst around when important policy decisions need to be made, since such policy decisions should be made and implemented by humans who can explain the confidence that exists regarding the knowledge that supports the given decision. The decision makers can then point to the analysts for the estimate of the degree of confidence that can be ascribed to the policy analysis that supports the decision. That’s how it’s supposed to work, and that philosophy is formalized in existing decision processes for complex technical systems such as transportation, telecommunications, power, military systems, etc.. You know, the important stuff….