[chatmasta]

This is similar to how our college CS problem sets were graded. We were given a spec, and we had to implement a program that conformed to it. We had access to 70% of the test suite during development, and another 30% was hidden and only evaluated after submission. We were graded out of 100.

It was effective at making you think about the problem and anticipate what tests might be missing. I can see how this would be effective for coding agents, which tend to get progressively lazier at writing tests as session context grows.

[seanmcdirmid]

The interesting twist here is that the test writer can also be wrong, the QA agent assigns blame when the implementation disagrees with a test case, and it’s not always going to be the implementation’s fault.

Makes me wonder if there should be problems for CS students where given a spec and an interface, but not an actual implementation, they write the tests instead.

[exe34]

Had this issue at uni back in '08 (physics), my code did what was asked of it, but the auto grader didn't like it. Nagged the professor until he had a closer look and fixed the issue, and asked me to resubmit.