[OrderlyTiamat]

> As far as I'm concerned the gold standard of understanding something is being able to construct a program that replicates it which is exactly what we can do with LLMs.

But we actually can't! We can build a program that can build a program that is the LLM, which is not the same! I'd argue that you're right insofar as training is concerned. We understand training very well. But the actual model, how it operates, what it actually knows, we don't know how to build that, we don't know what weights to put where.

[Izkata]

We actually have another example of this as well:

Malbolge is an esoteric programming language designed to be impossible to use. The first program written in it wasn't written by a human, it was written by another program.

But since then, with that working program to learn from, people have figured out how to write programs in malbolge: https://lutter.cc/malbolge/tutorial/cat.html