[6gvONxR4sf7o]

It's not about us being able to interpret answer or justification, but the reasoner's ability to justify. If a superhuman AI can justify its answers in terms of first order logic, for example, it could be defined as understanding the answers with respect to FOL. Whether we as humans are able to check whether this specific bot in fact meets that definition is a separate empirical question.

If that quant algo you mentioned just says "it'll go up tomorrow" that's different than "it'll go up tomorrow" with an attached "it's positively correlated with Y, which is up today" which is different from a full causal DAG model of the world attached, which is again different from those same things expressible in english. But again, those are definitions, which are separate from our ability to check whether they're met.

Luckily, we're not in the realm of bots spitting out unfeasible to check proofs, except for a few niche areas like theorem proving (e.g. four color theorem). For language models like in the article, the best I'm aware of is finding relevant passages to an answer and classifying entailments.

> A machine can't justify something you physically don't have enough neurons to comprehend.

We can't always verify its justification, but it either can or can't justify an answer with respect to a given justification system.