[thepasch]

> The algorithm is literally "predict the most likely next token".

That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."

[gmueckl]

Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).

[etskinner]

Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.