[federicof]

Hi, I understand your point. The problem with the feedback is that if we tell you for which inputs we're testing your solution, you could develop an optimized version of the solution for those special case, or worse you could precompute the solutions and just write a solution as a long if-then-else sequence. The purpose of the challenge is instead twofold: being able to understand the problem specs and being able two produce a generic solution that must work in an environment that you don't control (because that, at the very end, is what happens to the programs you write... They're going to be executed with inputs that aren't under your control and that you couldn't know in advance).

[tetha]

In theory, I agree with you.

In practice, I have spent an entire week during a universitary course with various problems from such problem site and eventually learned that the problem descriptios were different in a subtle way or certain edge cases were not considered or the test suites were flat out wrong. (This was particularly amusing, because they had added wrong test cases after certain solutions were accepted beforehand. Thus, It was clearly visible that the problem was solveable, because there were programs accepted).

Also I don't entirely agree with the argument that my program has to work in an unknown environment with unknown controls. In such a situation, I would at least get a logfile to see what was going wrong. Without that, debugging anything non-trivial is just not worth it without getting paid for it.

[federicof]

Ok but here the point is not about technology, it's very easy for us to give you access to the output of your program (in fact if you use a compiled language, and the compilation phase breaks, we show you the log). The point is that any kind of feedback that I give you access to that is generated from the input we send you to you program, will give you the chance to produce a solution that i not generic, because you know the input in advance, you can precompute the solution and just "print" it. You can object that we should generate the inputs randomly and on the fly. That can be definitely done for the very trivial problems (like reverse a string or a sequence) but will be impractical for more difficult problems that have very few input data sets that lead to a "meaningful" result.

Anyway, we're open to suggestions on how to improve the process. :)

[tetha]

Well you have a correct solver to the problem. Couldn't you just randomly generate an input, push it through the certified solver and supply me with the input-output-pair?

For one, even if this is the actual test input you use, I will not know it.

For two, it might not contain the trigger for my errors, but if I can get enough such samples, I should eventually get a sample which contains the problematic input.

For three, from what I see, the problems should be possible to generate randomly, especially if the certified solver rejects malformed input. E.g. for the missle problem, throw together N numbers and look what happens until the certified solver outputs a proper solution.

[federicof]

In theory it can be done, but it would be very expensive in terms of computational power. Anyway, you raised a very good point that we will keep in mind for the near future. Thanks indeed for your feedback and the nice discussion.

[tetha]

I see this concern. I thought a bit more about this during the day. The computational power required could be reduced if you stored the generated inputs and present them to the user if he requests additional help.

Given this, this system could be extended using a rating-system by users ("Click here if this input helped you solve the issue"). This would overall result in a bunch of inputs being generated (basically whenever someone needs a new input), and some edge case inputs will overall bubble to the top, because they are rated helpful more often than others.

Anyway, I'm happy to help. Good luck :)

[federicof]

Cool, this one is a very good idea actually, a user generated database of test cases for each puzzle! I'm putting it in our backlog, thank you very much!