[jhbadger]

The problem with that argument is it is trivial to write a Markov chain program that takes in text and then can generate the most probable series of words given a starting word. I myself wrote such a program in BASIC on a 64K 8-bit computer in the 1980s after reading one of A.K. Dewdney's columns. That wasn't at all an LLM though. There's a connection, sure, but one that is equating a paper airplane to a jet airliner.

[Charon77]

The issue with Markov Chain is you can't get good next token prediction on long enough context because once you see the last 1000 words instead of just 2, it's quite unlikely that your 'frequency' is populated for that exact combination, and markov chain don't work on token embedding that allows some encoding of meaning.

[AlecSchueler]

> and markov chain don't work on token embedding that allows some encoding of meaning.

Working on an "encoding of meaning" sure sounds a lot like reasoning.

[tanseydavid]

Consider the classic ambiguity example from Natural Language Processing: "Time flies like an arrow; fruit flies like a banana".

Markov chains are mostly about memorized local co-occurrence, while transformers are much better at context-sensitive disambiguation and compositional structure.

I do not have deep knowledge about these subjects but this is what I think was meant by "encoding of meaning" (LLMs). I also do not think this capability has to be thought of as "reasoning".