Hacker News new | ask | show | jobs
by botro 653 days ago
"The task consists of going from English-language specifications to Wolfram Language code. The test cases are exercises from Stephen Wolfram's An Elementary Introduction to the Wolfram Language."

I think this benchmark would really only tell me whether Wolframs book was in the training data.

2 comments

It's available online in HTML form, for free:

https://www.wolfram.com/language/elementary-introduction/3rd...

Yeah, may be should skip that benchmark.