[TuringNYC]

Thanks for the details. We're also trying to compare GPT 4 vs GPT 3.5 vs LLAMA2 and we've been putting together "exams" though one other thing on our mind is what happens when the next foundational model comes and scrapes our exams and the exam questions make their way into the training set.

This really does make me wonder about all the attempts to administer the USMLE to LLMs -- what are the chances the USMLE administered was uniquely created vs just put together from exam questions online? Ultimately no LLM is required if these exam questions are in the training set...just need a k-v lookup :-)