[FrustratedMonky]

>"at the expense of us actually not understanding how they reason with uncertainty"

That is the crux. It is doing something to anticipate words beyond the next token. It has to, to construct these long coherent documents. Just like humans do.

Just because we don't understand it doesn't mean it isn't reasoning. Isn't 'thinking ahead'.

Just like I can say we don't understand the brain, thus humans aren't actually reasoning.

There are a lot of brain studies that look into the pre-cursor changes in the brain pre-ceding conscious thought.

We can't say NN aren't doing something similar.

[js8]

> We can't say NN aren't doing something similar.

I agree, but my point was different. We can't say what it does theoretically, therefore we don't know how reliable it is (we don't understand the tradeoffs and failure modes). At least most humans have a way to assess their own reliability, and can see where their reasoning (or of their fellow humans) is inconsistent.

[FrustratedMonky]

There was good thread on this subject yesterday, more about LLM and Language.

It comes down to the physical world.

Humans can " assess their own reliability" when they can all point at something in the real world, and come to some agreement on what they are all seeing, what to call it, etc..

When humans get off base, if it is tangible, like an apple, they can all point at the apple, and bring themselves back into alignment, that is an apple.

But, for abstract concepts in philosophy, or morals, etc.. Something that is not tangible. Humans can 'drift' just as much as AI.

Humans can get into echo chambers -> and 'go nutz', absorbing others misinformation.

LLMs Learning from other LLSm' -> the 'models drift' over time.

https://news.ycombinator.com/item?id=37811610