[twayt]

I suspect you have adopted the speech patterns of people you respect criticizing LLMs of lacking “reasoning” and “understanding” capabilities without thinking about it carefully yourself.

1. How would you define these concepts so that incontrovertible evidence is even possible. Is “reasoning” or “understanding” even possible to measure? Or are we just inferring by proxy of certain signals that an underlying understanding exists?

2. Is it an existence proof? I.e we have shown one domain where it can reason, therefore reasoning is possible. Or do we have to show that it can reason on all domains that humans can reason in?

3. If you posit that it’s a qualitative evaluation akin to the Turing test, specify something concrete here and we can talk once that’s solved too.