[mcguire]

"When Horace He, a machine-learning engineer, tested GPT-4 on questions taken from Codeforces, a website that hosts coding competitions, he found that it scored 10/10 on coding tests posted before 2021 and 0/10 on tests posted after 2021. Others have also noted that GPT-4’s test scores take a dive on material produced after 2021. Because the model’s training data only included text collected before 2021, some say this shows that large language models display a kind of memorization rather than intelligence."

I'm sure that is just a matter of prompt engineering, though.

[COAGULOPATH]

But it got 10/10 on pre-2021 questions, with the same prompting method...