[zwaps]

This needs to be shown. For example, asking for something that is clearly in the training data (like Paul Grahams cv) is certainly not a proper way to test context recall

[jafitc]

Link from thread https://dev.to/zvone187/gpt-4-vs-claude-2-context-recall-ana...

[mejutoco]

Could we feed it Anna Karenina and ask it what is a difference between happy and unhappy families?

[jafitc]

Isn’t that the first sentence?

[mejutoco]

That is the point. Long book, checking the long context to see if remembers about the first sentence. Or you mean as a test it is better to randomly place the "needle"?

[zwaps]

It was trained on this book so again, this is not a good test

It will know the answer even without the book