[m12k]

So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?

[c7b]

The concept of predicting future elements in a series is not specific to CS. It's older than computers.

[kpw94]

Speculative execution techniques in software & hardware exist everywhere,

- Speculative multi threading

- Data Value Speculation

- Speculative Memory Disambiguation

- Runahead Execution

- Speculative Prefetching

- Multi-path (Dual-path) Execution (goes beyond branch prediction by computing both paths)

- Optimistic Concurrency Control (for database transactions etc)

[mike_hearn]

Maybe at very high level of abstraction, but there's no branching involved.

[lossolo]

Well, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.

[monster_truck]

Well, not exactly, but that was the dream we were sold (here be dragons)

[fragmede]

Well, the TPUs they're running on don't have branch prediction, so that had to end up somewhere in the stack.