[zarzavat]

> It's like you were in a multiple choice test and instead of picking which of answers A-D you think made more sense you picked E "I don't know".

This is a problem for testing humans too and the solution is simply to mark a wrong answer more harshly than a non-answer.

[zamadatix]

The technical solution is the easy half, the hard part is convincing people this is how we should be testing everything because we care about knowing the uncertainty in any test.

E.g. look at the math section of the SATs, it rewards trying to see if you can guess the right answer instead of rewarding admitting you don't know. It's not because the people writing the SATs can't figure out how to grade it otherwise, it's just not what people seem to care most about finding out for one reason or another.