This could be a solved problem. Come up with problems not online and compare. Later use LLMs to sort through your problems and classify between easy-difficult
Hard to do for an industry benchmark since doing the test in such a mode requires sending the question to the LLM which then basically puts it into a public training set.
This has been tried multiple times by multiple people and it ends up not doing so great over time in terms of retaining immunity to “cheating”.
This has been tried multiple times by multiple people and it ends up not doing so great over time in terms of retaining immunity to “cheating”.