[parineum]

> If they were trained to answer “I don’t know”

If they were trained that an answe of "I don't know" was an acceptable answer, the model would be prone to always say "I don't know" because it's a universally acceptable answer.

It's a better answer even if it does "know".

[chongli]

That could be fixed with the right scoring scheme in training. The SAT exam (for college-bound high school students in the US) used a scheme like this for multiple choice questions. Correct answers are awarded 3 points (with choices a,b,c,d), incorrect answers are penalized with -1 point, and leaving the answer blank (equivalent to "I don't know") is worth 0 points. This way, the expected value of guessing a random answer when the student doesn't know is 0 points so you might as well leave it blank if your confidence in the answer is no better than a random guess.