[moralestapia]

>the quintessential language model task of predicting the next word?

Based on what? The whole test is flawed because of this. Even different LLMs would choose different answers and there's no objective argument to make for which one is the best.

[sorokod]

The one provided in the original post.

[moralestapia]

I don't see any of that.

Quote?

[JoelEinbinder]

The prompts you see in the quiz are from real hacker news comments. Whatever word the commenter said next is the "correct" word.

[moralestapia]

This is what I see,

  Are you smarter than a language model?

  There are a lot of benchmarks that try to see how good language models are at human tasks. But how good are you at the quintessential language model task of predicting the next word?

And then a list of questions.

How am I supposed to know it has anything to do with HN?

[JoelEinbinder]

After the quiz, the source is linked along with the full comment.