Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct.
A regression test is checking causality: Changes in new code, updating dependencies, updating the OS the software is running on, updating shared libraries, porting the code to a new platform, etc. aren't supposed to change the test results.
"I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"
> "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"
Cosine is a terrible example to use for that idea. It's pretty likely to change, for certain x, in similar circumstances to your examples of "when test results should never change".
If it's likely to change, then you especially want the regression test so you can decide how to handle the divergence during your port. Maybe one library preserves the signal on NaNs and the other doesn't. Or maybe the CPU's default rounding mode is different when called in this context, and you're off by 1 ulp.
In either case, if the behavior is to change, it should change as an informed decision and not because nobody noticed.
This looks similar to snapshot testing in UI, where you save an output of UI components and test system notifies you when the output changes. This can be useful to detect changes in components that you didn’t intend to change.
- This here is akin to UI testing framework X where the test framework can compare an expected screenshot of the UI to a screenshot of the actual UI!
The last one basically requires automation if you want anyone to make use of it. The regression testing automation described in the OP is a nice-to-have, not a so-good-that-it-gets-a-new-name.
"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test.
https://en.wikipedia.org/wiki/Regression_testing
That’s fine and I have no objective argument against it. But I don’t see much reason to need two different names for tests that do the same thing merely based on how they were introduced. Sometimes I add a regression test because I fixed a bug, and sometimes I add a regression test because I just implemented a feature that I don’t want my future self to ruin: six months from now they will co-exist in the same suite and serve the same function.
One reason to call bug fix tests for “regression tests” (and only those kinds of tests) is that someone might regress the code base through a merge conflict (maybe they effectively undo a commit?). So that’s one argument I suppose.
"I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"