[praptak]

This would probably produce a program that cheats a lot. Perhaps not by simply hardcoding the answers to the tests, but by finding a good compressing function for the set of (input, output) pairs in the test. I bet this would seldom provide the expected answers on inputs not covered by the tests.

[wlievens]

I second this. Overfitting can be a problem even in human-written test scenario's. Usually overconfidence in your test suite is a cause.