[leetcrew]

as a TA, i love when professors set up grading scripts for me, and as a student, i love knowing what grade i will get before turning in the assignment. unfortunately, you need the students to have the basic level of competency where they can actually write code to a testable interface or correctly use print statements to produce good enough output for a script to grade by parsing. in the course i currently TA, neither of these prerequisites are met. many students turn in programs that will not compile and i am expected to give generous partial credit if i can determine that they are "most of the way there".

i also have pedagogical concerns about providing the test script ahead of time. i suspect that it biases low effort students even more in the direction of "guess and check" rather than the ideal (imo) "reason and predict".

all that said, the approach you outlined is probably the best feasible approach for most cs courses.

[ngomez]

The issue of incentivizing students to guess-and-check when providing the test scripts upfront is, IMO, fixed by making the students write the tests themselves. This paper explains it pretty well:

https://www.cs.tufts.edu/~nr/cs257/archive/stephen-edwards/a...

Essentially the students would write a test suite and the grading framework would grade based on

1) Code coverage when running the student's test cases against an instructor's reference solution

2) Correctness of output: running student's test cases on student's code and comparing with output from running those test cases on the reference solution

3) Number of test cases passed in student's test suite

Also from the paper:

"All three measures are taken on a 0%–100% scale, and the three components are simply multiplied together. As a result, the score in each dimension becomes a “cap” for the overall score—it is not possible for a student to do poorly in one dimension but do well overall. Also, the effect of the multiplication is that a student cannot accept so-so scores across the board. Instead, near-perfect performance in at least two dimensions should become the expected norm for students."

Students still get the benefit of knowing their grade when they submit, and as an added bonus students get more hands-on experience with test-driven development. Having the students write the tests themselves also increases the cost of mutating code until it just works.

[leetcrew]

this sounds like a great solution. i particularly like that it encourages careful thought about test cases.

[emidln]

It sounds like students who can't program need to be offered/forced into a 099-level class on whatever language or languages are being used for pedagogical purposes prior to actual CS classes. We don't let students who can't handle Algebra and Geometry take Calculus classes and we shouldn't suffer students who can't generate compiling code in CS classes. I made most of my money in college because such a class didn't exist.