[lolinder]

We're focused on the bad example because it's literally the title of the article and the model's inability to solve that problem has nothing to do with context windows and everything to do with "when all you have is a hammer".

It doesn't matter if the context window is large or small, the Harry Potter Problem as formulated is going to be just as hard because it's not a problem with false advertising in context window sizes, it's a problem inherent to the computing paradigm.

A version of the Harry Potter Problem that was formulated around a model's ability to recall specific scenes of a novel would be much more useful as an illustration of the limitations of the supposedly-large context windows.

[araghuvanshi]

Well the same principle of false advertising re: context window sizes also applies to its inability to count, no? AI companies claim that their models can do math, so wouldn't a regular developer assume that they can also count?

And if I can't trust a so-called SOTA model to partially answer - say, recall each mention of the word "wizard" instead of just giving me the wrong answer - then why should I trust it to list out specific scenes? That's even harder to benchmark.