|
|
|
|
|
|
by ben_w
11 days ago
|
|
|
The infinite monkey theorem assumes random distribution of symbols*. Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct. Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart). * Actual monkeys not being like this is, while amusing, irrelevant ** https://talkie-lm.com/introducing-talkie |
|
|