[visarga]

I think of syntax as applying a rule. A program is just syntactic operations. The idea is that it has two aspects: "program in execution" and "program as data". Rules expressing behavior and rules expressed as data, in which case it can be processed by other rules.

One concrete example is a bootstrapped compiler. It is both data and execution. It can build itself, putting its output as input again. Another example is in math - Gödel's arithmetization, which encodes math statements as numbers, processing math syntax with math operations. And of course neural nets, you can describe them as purely syntactic (mechanical) operations, but they also update rules and learn. In the backward pass, the model becomes input for gradient update. So it is both rule and data. DNA too.

These systems that express rules or syntax that is adaptive, I think they make the leap to semantics by grounding in the outside environment. The idea that syntax is shallow and fixed is wrong, in fact syntax can be deep and self generative. Syntax is just a compressed model of the environment, and that is how it gets to reflect semantics.

This was an argument against Stochastic Parrots and Chinese Room (syntax is not sufficient for semantics) maxim. I aimed to show that purely mechanical or syntactic operations carry more depth than originally thought.