[Syzygies]

One can't have an informed opinion on this without witnessing how grading a college exam takes place, first hand.

Grading a stack of blue books is a "just kill me now" brutal experience. A majority of cognitive effort is just finding the page for the next problem to grade, with finding the answer another delay; any programming language with this kind of access latency would stay a "one bus" language. So of course professors rely on disinterested grad students to help grade. They'll make one pass through the stack, getting the hang of the problem and refining the point system about twenty blue books in, but never going back.

With stapled exams one problem per page one can instead sort into piles for scores 0-6 (if you think your workplace is petty, try to imagine a sincere conversation about whether an answer is worth 14 or 15 points out of 20), and it's easy to review piles.

When I had 200 linear algebra exams to grade at once, I'd scan everything, and use my own software to mark and bin one question at a time, making review passes a pleasure. I could grade a 10 question final in one intense sitting, with far more confidence in the results than team grading ever gave me.

[ipcress_file]

I think problem number one is that you had 200 exams to grade.

[kkylin]

This is not an uncomon situation in e.g., math departments across the US. Luckily I've not had to grade more then 60-70 in a single term, or ~200 but a select number of problems in a large course.

Also, second stapled exams with prescribed spaces for answers. So much time wasted just looking for where the answer is.

[downrightmike]

The children yearn to learn. And the admin cuts costs.

[8bitsrule]

How else can they hire more admins?

[spicyusername]

If learning is the goal, and we just learned that we need to do more work to provide an experience conducive to learning, then the cost for learning just increased.

First, its interesting to see a situation where a new technology INCREASED the cost of something.

    Grading a stack of blue books is a "just kill me now" brutal experience.
    When I had 200 linear algebra exams to grade at once

Second, this is alluding to one of the dimensions along which that cost is going to be born, staffing. If its too many papers to grade, then more staff are needed to grade them.

We're probably not going to get that, though, and instead the cost increases, the funding stays the same, so the quality will decline to compensate.

So it goes...

[armchairhacker]

It doesn’t have to be a book, it can be a cheap laptop without internet.

[teeray]

The idea is the same: society needs AI-free enclaves to measure aptitude. How those are realized can vary, of course.

[smelendez]

Probably someone will build and sell a dirt cheap laptop with nothing on it except a basic word processor for this purpose.

It’s still a hard problem though. If the students have the laptops outside of the testing site, they can load cheating materials on them, or use them to smuggle questions and answers out if the test is used in multiple class sections. You realistically will not lock down a laptop students can take home sufficiently that some people won’t tamper with it.

Otherwise you have to have enough laptops to get each student a wiped and working machine for every test, even with lots of tests going on. And students need to be able to plug them in unless the batteries are rigorously tested and charged, but not every classroom has enough outlets. And you need to shuttle the laptops around campus.

Then you need a way to get the student work off the laptops. You probably want the student to bring the laptop up when done and plug in a USB printer. Anything else, like removable media, and you have to worry about data loss or corruption, including deliberate manipulation by students not doing well on the exam, and students claiming what got graded wasn’t what they meant to hand in. And you still have to worry about students finding a way to deliberately brick their laptops and the inevitable paper jams and other hardware failures, especially an issue when students need to leave on time to get to another class.

So you need systems that are cheap but reliable, tamper-resistant but easy to diagnose and maintain, robust against accidental data loss but easy to reliably erase, able to export data and install security updates without letting students surreptitiously input data, and heavily locked down while easy to use for students with a wide variety of backgrounds, training, and physical abilities.

[armchairhacker]

Store in each classroom a custom-made rack for the laptops (with at least one laptop per seat plus backups). Distribute the laptops at the start of exams and collect them at the end.

The rack charges the laptops, streamlines distributing/collecting tests, prevents tampering, and reports defects. A professor uploads the test to their LMS and specifies the exam time-frame and location. Before the exam, the LMS transfers the test to the correct rack, which saves it to the laptops. After the exam, the rack loads the student responses and transfers them to the LMS for the correct class. Each laptop has a light under it whose color indicates whether it's in standby, waiting to be distributed (shortly before exam start), has a submitted test, or is malfunctioning (the rack periodically pings laptops in storage to verify they still work). Multiple professors can queue exams to the same location, in different time-frames, and the LMS and rack know when to prepare each test on the laptops.

The hard part is to implement this well. The rack's hardware and software must be reliable; nonetheless it will fail (hardware breaks), and exams get moved and rescheduled, so there must be a way to transfer tests to another classroom and time-slot. The laptops can have slightly less reliability, since there are backups; but failures can't be clustered, and if a laptop is transferred from another rack during an exam, it should download the destination rack's test and become active (so if one rack's backup laptops run out, laptops can be borrowed from other classrooms). The laptops must periodically backup in-progress tests using wifi or bluetooth, and if a laptop breaks during the exam, the student can resume their progress on another. Tests must be downloaded well before exam time, in case there are problems (laptops should be able to store and queue multiple tests). Laptops must handle exams that start late (up to the official exam end time) and end late (including overlapping the next exam's start time, in which case the next exam is loaded when the laptop is put back into the rack). The rack must absolutely not indicate that a laptop is ready with an exam if it's not; and (since tests are downloaded before exam time) nothing unique should happen at exam start time (to reduce last-minute surprises, and let a professor who isn't convinced take out a laptop and check that it's functional 15 minutes before). Last but certainly not least, the UI to submit tests and grade responses should be intuitive, simple (not overwhelming) yet powerful (can handle edge-cases like rescheduled tests, randomized tests, accommodations; or be extensible enough to support these features).

Despite all the above requirements I actually think such a product is feasible. Unfortunately, with firsthand experience of typical EdTech and academic bureaucracy (and I'm not a professor so I don't know the worst of it), I'm skeptical it would be adequate quality unless the company designing it is uniquely motivated and capable.

[smelendez]

Yeah, it’s implementing it well that’s the issue. And securely: Now the laptops have Bluetooth and WiFi, and you can assume some talented and obsessive CS students have access and are regularly finding ways to jailbreak them.

Also, having classrooms full of laptops full of tests assume classrooms are secure and at many colleges they’re unlocked by default. There could be a 9am class and exam, then the room is empty until 11, then the same thing until a class at 2 and a night class at 6, and at 8 the PenTesting and Lockpicking Club meets in the room.

Keeping dozens or hundreds of laptops in every classroom also dramatically shifts the reward for burglars which colleges will not like.

[pimlottc]

Or get a stack of dirt cheap blue books and pens.

[Balgair]

You make a great point here. The quality of the grader may be higher with an AI than a stressed out grad student.

It's long been known that a longer essay answer is more likely to get a higher grade. And yeah, having been a student and a stressed out grad student, after about the 20th exam, only length is the real signal of grade.

Other comments point out that with the prices of tuition these days, students should be expecting a lot higher quality of feedback (grading) than what they are getting at any random R1.

It really does seem that the University system (as opposed to the college-esque system) is broken and that the additional AI fears are just another log on the already collapsed bridge. We're getting over the wrong thing.

[mlpoknbji]

This problem is solved by software like gradescope. Makes grading extremely fast and much more consistent (because of easy rubric adjustments on the fly). This is more concerning a STEM exam, admittedly I don't know how well this works on humanities essays.

[kkylin]

Gradescope partially solves the problem. There are certain kinds of questions that are more amenable to rigid rubrics (e.g., techniques of integration in a beginning calculus course) where the space of correct solutions is rather limited. Once the material gets more advanced, the space of correct (and partially correct) solutions gets really large, fast. One could design exam questions to restrict the solution space, but sometimes at the cost of testing real understanding. Also, at that point giving partial credit with Gradescope-style rubrics is not always straightforward (not impossible, but not trivial).

[kjkjadksj]

Speaking as someone who has graded in grad school, the general advice is to look at a few questions before you start dishing out points so you can refine your internal rubric. Going back is also done because that is easier than having a student show up at office hours and ask why sally only got marked down 3 points for the same mistake on question 5.

[euroderf]

If you try to blow thru all the books in one pass, it's grossly unbalanced for the head and tail, and many books in-between.

I did it in three passes - which might be all in one session or might be broken into smaller sessions. First, skim them all for spelling & grammar mistakes, with a red pen. Second, read them all (but not too deeply) to identify where they make key points, with a green pen. At this point you have a really good handle on the general level of accomplishment. Third, take a blue pen and comment them and grade them.

[Mountain_Skies]

One of my history professors would watch horror movies while grading history essay exams. Have to give him some credit for not taking the easy way out with scantron multiple choice test, requiring long form essays to demonstrate understanding of not just historical events but also their impact.

[hollandheese]

Huh? Blue books are typically used for essays and are graded whole.

I've never seen anyone attempt blue books for anything else.

[SoftTalker]

Late 1980s blue books for almost everything that wasn’t multiple choice. The other style was multiple pages, with large “show your work here” areas below each question.

[rubidium]

I used them in graduate physics courses 2008-2010

[ramenbytes]

And I used them in a graduate physics course last fall.