[TechBro8615]

I had one professor who used private (30%) and public (70%) tests. You could iterate on the public tests up until the submission deadline. But you never saw the private tests until the professor ran them on your program after you submitted it.

I think this was a good middle ground that forced you to think beyond the rigid parameters of the public tests, and discouraged things like “switch” statements.

Quality of code was also a component, as determined by a TA.

[renodubois]

I graded for a class that took a similar approach to this - the students had a small amount of tests available to them that allowed them to iterate over the requirements for the assignment, but we had more exhaustive tests that we ran automatically. In addition to checking test completeness, we also looked at the source code itself and graded for quality of that. The automated runner handled a lot of the hard work for us, but it definitely wasn't a situation where we saw some failed unit tests and then proceeded to fail them or anything.

The system used to pull this all off is pretty interesting, I think. It's all open source: https://github.com/redkyn

[aroch]

Ha, I did something very similar also with Gitlab and runners. I was always amazed at the new and creative ways students managed to break the autograders we wrote -- particularly for our BASH scripting assignments.

I was also never brave enough to have my autograders submit grades directly to Canvas after the first time I did it. I confidently had it directly submit grades the first week during the first year I implemented autograders. Half the class forgot a shebang on their shell script, and the autograder happily gave them a 0.

For bonus fun times: Gitlab's runner timeout is pretty much nonexistent if you're using the shell runner. It just won't kill them if they take too long, so I had to write our own management system for runners. Silly university IT not support containers :(

[3PS]

One of my favorite classes had a few simple tests provided by the professor, and the rest of the tests were provided by students. Our grade was based on our score on the overall test suite, combined with the quality of our own provided test case, our code quality as judged by a TA, and an external writeup. This was fun because it encouraged students to create brutally hard tests to cover all sorts of corner cases in a way that was openly available to everyone.