[thewataccount]

IDK if it helps but I find these examples interesting, predict the most likely word for the following phrases (whatever immediately comes to mind first):

- hello! How are

- What is 1+1?

- Today is Monday. That means yesterday was

- My favorite color is

Humans will likely estimate these as "you", "2", and "sunday". For the last one, there's not enough context to estimate the next word, but we will almost certainly estimate it as a "color", and might just have to guess at random "red"/"blue"/"orange"/etc, so there can be multiple possible answers. If you've seen enough data you might know that a specific color like blue might be more frequently mentioned in this context, and would be more included to guess "blue" because of that.

1+1 is interesting because most humans can immediately estimate 2. However given the question "What is (3.21^7)/(3+6.4*11)?" we can no longer quickly estimate the answer, we would have to manually (even if mentally) calculate the answer. I think this closely parallels how our current LLMs can do basic math they've seen a lot, but fall apart on more complex math since they aren't able to do the actual calculations and are forced to estimate.

"Today is Monday. That means yesterday was" is interesting because this would appear to imply that there is "logic". That you listed out the days of the week, looked at monday, and chose the day before. However, most likely you "just knew" it was most likely sunday given your previous knowledge. The LLM is unable to actually compute the answer, but it can estimate it based on it's training data.