Yeah in their Fibonacci example if it printed out 510 instead of 610 you'd still have a bug and think you had tested it. Especially confusing for future people who will assume it works because there are passing tests!
How do you know if you have the right result though? You might know if you have a plausible result. Like if it output -1 then you know something is wrong I guess.
There's a much higher chance of detecting bugs that give plausible output if you aren't given the opportunity to say "eh looks plausible I won't bother double checking it".
Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place. If this gets the friction of writing a test at all so close to zero that these programmers start writing tests (albeit sometimes blindly accepting the output), then it's better than just trying their program on some inputs and calling it a day. It writes down the current output of the program. That's a big step up already. Now people evaluating the code can read some of its outputs without downloading anything.
I personally already use a similar cycle to expect-test when I write tests. A great place to start when writing test assertions is the debug output, just like this thing uses. Then you convert the output into assertions after you have thought through which parts are right or wrong. Just like you can do with expect-test, but without the automation. If you don't know whether the output is right or not, just add an assert(false, "hmm, not sure about this") aka todo!() and voilà, your test fails and future you can be prompted to check over it again.
Sometimes the output is obviously wrong, but you still don't know what the right output is. (At this point you know you're doing useful work!) The remedy is the same. Just make the test fail somehow.
> Any programmer dumb enough to just blindly accept that their program is correct is also a dumb enough programmer not to have begun writing a test in the first place.
Then what's the point of this methodology? It requires you to write tests and also blindly accept that your program is correct.
Maybe they should just rename it to "plausibility tests" or similar because that's what they're really testing. And while that does have some value, I think most of the value is negated by the fact that it sounds like they are properly vetted tests which they are not.
So a more appropriate name would help a lot. I still think it's a bad idea though.
> It requires you to write tests and also blindly accept that your program is correct.
No. You can say no. Just don’t accept it. You’re a human and it asks. Even if you do accept it you can modify it because you have eyes and a keyboard and it’s written right there where you wrote your test.
As I said before, it's probably better than nothing in that it will help you detect obviously implausible results. But it really needs to be labelled as such otherwise people will assume that these are properly curated "golden" tests.
It's a repl, so you build the final output incrementally. Testing becomes part of the development workflow like you would do in languages that rely on the repl like lisps.
For example, you start with the inputs and you apply the first layer of transformations, then check what it does makes sense. Then maybe you refactor it out in its own function and add the generated test for it. Then you move on the next step and so on until you have the final result.