[addcn]

For sure. Great argument

+ the experiments may already be in the dataset so it’s really testing if it remembers pop psychology

[a123b456c]

Yes. A stronger test would be guessing the results of as-yet-unpublished experiments.

[jslezak]

They did this. Read the paper

[abcgde]

Well, they looked at papers that weren't published as of the original model release. But GPT very likely had unannounced model updates. Is it not possible that many of the post 2021 papers were in the version of GPT they actually worked with?