[ineedasername]

When were the GPT-4 benchmarks calculated, on original release or more recently? (curious per the debate about alleged gpt-4 nerfing)

[lappa]

They're based on the original technical report.

"Refuel" has run a different set of benchmarks on GPT-3.5 and GPT-4 and found a decline in quality.

https://www.refuel.ai/blog-posts/gpt-3-5-turbo-model-compari...

[ShamelessC]

Plenty of the complaints/accusations predate the release of the 0613 set of models.

To be clear, I have trouble with the theory as I have not yet seen evidence of "nerfing". What you provided is actually the _only_ evidence I've seen that suggests degradation - but in this case OpenAI is being completely transparent about it and allows you to switch to the 0314 model if you would like to.

Every complaint I have seen has been highly anecdotal, lacking any rigor, and I bet are explained by prolonged usage resulting in noticing more errors. Also probably a bit of "the magic is gone now" psychological effect (like how a "cutting edge" video game such as Half-Life 2 feels a bit lackluster these days).