[botro]

"The task consists of going from English-language specifications to Wolfram Language code. The test cases are exercises from Stephen Wolfram's An Elementary Introduction to the Wolfram Language."

I think this benchmark would really only tell me whether Wolframs book was in the training data.

[bufferoverflow]

It's available online in HTML form, for free:

https://www.wolfram.com/language/elementary-introduction/3rd...

[smusamashah]

Yeah, may be should skip that benchmark.