[dshields1]

> The AI isn’t used to directly grade the papers; rather, it turns grading into an automated, highly repeatable exercise by learning to identify and group answers, and thus treat them as batches.

It seems like the AI is identifying equivalent answers among respondents. So if you mark an answer correct on one test, every other test with the same answer will be marked correct. I worked for a small competitor of this product in college and we had a lot of trouble with this problem, especially with answers that were prone to spelling mistakes, or could be written in many ways. Kudos to them for doing this well.

Another fun bit of AI in this space is in identifying where the answer key might have made a mistake. We developed some algorithms for determining the most likely answer to a problem given the responses. We never released it but I worked on a tool that would grade tests without an answer key at all. Using 50 question tests in a few freshman physics classes I was able to get the right answer a little over 97% of the time.

[analog31]

This makes a lot of sense, and follows typical methods that were done by hand long ago. When I TA'd freshman physics at a big university in the 90s, the grading session was a two-pass assembler. First pass, we simply identified the possible answers. Second pass, we marked them. This gave us much more consistency, and it was quicker overall.

Amusingly, the exams always asked for a numerical answer, from which we could guess which mistake they made, then we would find that mistake in their calculation and mark it. Without that trick, identifying the specific mistake in each answer was a pretty tedious process.