[moyix]

> I think there is a very fine line between claiming that a CNN's weights represent an algorithm and that its weights can be _interpreted_ as an algorithm.

Yeah, I agree this is an issue. It feels a bit reminiscent of Searle's Waterfall argument, and so I'm inclined to turn to Scott Aaronson's response here [1; Section 6] – basically, how much work is the interpretation itself doing? If you actually tried to use the "algorithm" to do what your interpretation says it should do, how much work would you have to put into that mapping? If the work required amounts to just implementing the algorithm and effectively ignoring the CNN (or waterfall), then the interpretation is what was doing all the work.

IMO the Curve Circuits case passes this test, since they show that you can mechanically take out the learned weights, drop in the weights derived algorithm they reverse engineered, and the model still works about as well.

> In any case, I would like to understand how a language model can represent an algorithm.

Likewise! :)

[1] https://www.scottaaronson.com/papers/philos.pdf

[YeGoblynQueenne]

Thanks for the link to Scott Aaronson's paper, that I hadn't read. I think your comment helped clarify something that bothered me with the CNN paper: I would feel more convinced if the claimed algorithm had been implemented _without_ a neural net. What the authors did was manually set the weights of another neural net. If an algorithm is discerned in the neural net's weights, then why can't it be programmed in the usual manner, in some computer language? If it can't, then that's very difficult for me to accept it as an algorithm, because I can't see what it is, exactly. The authors claim that the algorithm can be described in a few English sentences, but in truth these few English sentences are a high-level description of the behaviour of the CNN rather than a sequence of instructions (which is what I have in mind, perhaps erroneously, as an "algorithm").

I'm not necessarily asking for simplicity. I'm used to algorithms being relatively simple things that can be implemented in a few lines of code and I'd think of something more extensive as a "program" rather than an algorithm, but I appreciate that an algorithm encoded in the weights of a deep neural net could be something really big. I just want to see this algorithm written down in pseudocode at least, in a form that can be executed by a device other than a neural network (like me, following the pseudocode). I think that is the opposite of Aaronson's point actually.

I think I might be a realist:

https://youtu.be/STFcvzoxVw4?t=97