Hacker News new | ask | show | jobs
by jbellis 16 days ago
It tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919
7 comments
krackers 16 days ago
training on the test set is all you need
link
ch4s3 16 days ago
It's ready for law school!
link
fxwin 15 days ago
looks like they have since fixed the dataset, hope they release an updated report (either without the leaked evals or a new checkpoint/weights)

https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f9...

link
sva_ 16 days ago
https://nitter.net/eliebakouch/status/2077425801633427919

That is very disappointing.

link
Tostino 16 days ago
That's just embarrassing. It keeps happening though.
link
amai 15 days ago
It seems they corrected this already: https://nitter.net/jphme/status/2077468512025477226#m
link
jbellis 15 days ago
I guess you didn't read ellie's reply directly underneath that?
link
istvan0 15 days ago
Reminds me of: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
link
AtlasBarfed 16 days ago
scrape the scrapers... all's fair...
link
sajithdilshan 16 days ago
Disappointing. Sounds like Volkswagen 2.0
link