>Argument from ignorance, also known as appeal to ignorance (in which ignorance represents "a lack of contrary evidence"), is a fallacy in informal logic. It asserts that a proposition is true because it has not yet been proven false or a proposition is false because it has not yet been proven true.[0]
Why should we assume that the training set includes the answer key in the absence of evidence?
We know that the training set is large and non public, and that answer keys are generally available and ML likes to output exact answers when it's been trained on the questions.
It's significantly more likely that a model outputting the right answer was trained in that answer than not
Still questions that I would assume are semantically similar to the questions you can find in exam prep material all over the internet. My point is that exams are a crutch we use to determine how well a person studied a subject. A crutch we use because we seem to lack better measuring devices. It's very much possible to ace an exam while at the same time being horrible at actually applying/working on a subject. I'd argue therefore that measuring how well LLMs perform on exams designed for humans is simply a more complicated Turing test, with all its shortcomings.
Why should we assume that the training set includes the answer key in the absence of evidence?
0. https://en.wikipedia.org/wiki/Argument_from_ignorance