[leetcrew]

as a grader, it is already hard enough to keep up with the work that gets generated by thirty students attempting the same weekly homework assignment. i don't know how i could possibly grade thirty different projects in a week, let alone in any fair or reasonable way.

i've always wondered why they don't do something like select a random number of students and ask them to explain their work in an interview with the professor or TA.

i figure, even if they cheated, if they can give a satisfactory explanation, they probably learned the damn thing anyway. plus, you don't have to select even half the students before word gets around that they better understand the work they are turning in.

[emidln]

Add a grading section to each assignment with something like this: --------- Program is expected to maintain this interface <link to interface here> over stdin/stdout (or sockets, etc).

Students will pass when their program passes the grading/test suite. Test suite can be check ahead of time by uploading binary/zip/tarball to this location using their student id: <location here>. Detailed instructions <here>.

IIRC, this is what my university did 15 years ago. Not everyone had unique homework, but often the homework assignments were not all the same. I guess how unique they are depends on if you have some parameters in your homework generation or if you have a bunch of misc grad students to generate a few sets of homework that you can build up over time.

How might you grade style? IMO Don't. If you must, then save it for interactive review/lab sessions/office hours or style-specific spot checks (e.g. "At least once per semester your assignment will be additionally graded on program style according <insert guidelines here>. This will be added as additional points to your semester total.")

[leetcrew]

as a TA, i love when professors set up grading scripts for me, and as a student, i love knowing what grade i will get before turning in the assignment. unfortunately, you need the students to have the basic level of competency where they can actually write code to a testable interface or correctly use print statements to produce good enough output for a script to grade by parsing. in the course i currently TA, neither of these prerequisites are met. many students turn in programs that will not compile and i am expected to give generous partial credit if i can determine that they are "most of the way there".

i also have pedagogical concerns about providing the test script ahead of time. i suspect that it biases low effort students even more in the direction of "guess and check" rather than the ideal (imo) "reason and predict".

all that said, the approach you outlined is probably the best feasible approach for most cs courses.

[ngomez]

The issue of incentivizing students to guess-and-check when providing the test scripts upfront is, IMO, fixed by making the students write the tests themselves. This paper explains it pretty well:

https://www.cs.tufts.edu/~nr/cs257/archive/stephen-edwards/a...

Essentially the students would write a test suite and the grading framework would grade based on

1) Code coverage when running the student's test cases against an instructor's reference solution

2) Correctness of output: running student's test cases on student's code and comparing with output from running those test cases on the reference solution

3) Number of test cases passed in student's test suite

Also from the paper:

"All three measures are taken on a 0%–100% scale, and the three components are simply multiplied together. As a result, the score in each dimension becomes a “cap” for the overall score—it is not possible for a student to do poorly in one dimension but do well overall. Also, the effect of the multiplication is that a student cannot accept so-so scores across the board. Instead, near-perfect performance in at least two dimensions should become the expected norm for students."

Students still get the benefit of knowing their grade when they submit, and as an added bonus students get more hands-on experience with test-driven development. Having the students write the tests themselves also increases the cost of mutating code until it just works.

[leetcrew]

this sounds like a great solution. i particularly like that it encourages careful thought about test cases.

[emidln]

It sounds like students who can't program need to be offered/forced into a 099-level class on whatever language or languages are being used for pedagogical purposes prior to actual CS classes. We don't let students who can't handle Algebra and Geometry take Calculus classes and we shouldn't suffer students who can't generate compiling code in CS classes. I made most of my money in college because such a class didn't exist.