[solomonb]

I gave chatgpt 3.5 the type signature for a co-algebraic encoding of a mealy machine:

    newtype Mealy s i o = Mealy { runMealy :: (s, i) -> (s, o) }

And it gave a really impressive analysis.

Then I scrambled all the names and asked with a fresh context like:

    newtype Foo z e g = Bar { blob :: (z, e) -> (z, g) }

It got completely confused and generated a bunch of non-sense. It was at that moment I realized that LLMs don't really understand anything.

And yes I understand that a newer model would not get confused by this.

[bonoboTP]

In high school math class our teacher swapped out all the symbols in the epsilon delta definition of limits, and asked us what this equation expresses, and many students struggled to interpret it.

I don't think this test shows that an LLM doesn't "understand". It shows more that it has similar failure modes as humans.

[solomonb]

Well first of all I think there is more implicit data encoded in the symbols of the epsilon delta definition of limits. In the Mealy example they really just labels for arbitrary sets. The LLM actually failed a much simpler relabeling exercise. Setting that aside, I still think the analogy is flawed.

The student is mid learning process and its entirely reasonable for them one to be relying on pattern recognition until they have fully internalized the subject. The model is fully trained and should thus have internalized their understanding of the subject.

Additionally the student can update their understanding when pattern recognition fails. The model is fully cooked and will never do more then pattern recognition.

[pixl97]

>The model is fully cooked and will never do more then pattern recognition.

The correct answer to this is "Thank fucking god".

Humanity as a whole is not ready for continuous learning models. They'd either learn very fast and actually place people at their jobs and cause mass hardships for humans, or they'd go off the alignment rails really fast and cause mass hardships for humans.

Hopefully we'll get a nice steady onramp to continuous learning where we can iron these issues out.