[beering]

> In my mind, understanding a thing means you can justify an answer.

What if the language model can generate a step-by-step explanation in the form of text? [0]

There's no guarantee that the reasoning was used to come up with the answer in the first place, and no proof that the reasoning isn't just the product of "a really fancy markov chain generator", but would you accept it?

We're really walking into Searle's Chinese Room at this point.

[0] https://nitter.hu/kleptid/status/1284069270603866113#m

[naresh_xai]

Umm, no there are clear verification methods for Explainable AI techniques today. One way to check the justification would be if things which were important in the justification were removed in some sense, then would the output change signficantly. Sort of like a sensitivity test for justification.

[6gvONxR4sf7o]

Searle's Chinese Room is exactly why I started thinking of understanding this way. It convinced me that a one-size-fits-all notion of understanding isn't useful. But it also made me think that understanding "with respect to system X" is useful.

If you can challenge an answer and get justification expressed in the form of X, then it understands with respect to X. A step-by-step text explanation is one form of X.

> ... but would you accept it?

This is all to sidestep questions of whether you accept X as "real" understanding or not. :D

[piyh]

>There's no guarantee that the reasoning was used to come up with the answer in the first place, and no proof that the reasoning isn't just the product of....

You're holding machines to a higher standard than we hold people.