[jsharf]

I'm not anti-connectionist, but if I were to put myself in their shoes, I'd respond by pointing out that in E=MC^2, C is a value which directly correlates with empirical results. If all of humanity were to suddenly disappear, a future advanced civilization would re-discover the same constant, though maybe with different units. Their neural networks, on the other hand, probably would be meaningfully different.

Also, the C in E=MC^2 has units which define what it means in physical terms. How can you define a "unit" for a neural network's output?

Now, my thoughts on this are contrary to what I've said so far. Even though neural network outputs aren't easily defined currently, there's some experimental results showing neurons in neural networks demonstrating symbolic-like higher-level behavior:

https://openai.com/blog/multimodal-neurons/

Part of the confusion likely comes from how neural networks represent information -- often by superimposing multiple different representations. A very nice paper from Anthropic and Harvard delved into this recently:

https://transformer-circuits.pub/2022/toy_model/index.html

[ctoth]

Related: Polysemanticity and Capacity in Neural Networks https://arxiv.org/abs/2210.01892