|
We're in general agreement about the purpose of unit tests. I disagree on a couple of points. Tests do not document the API. No test is complete, and for that reason alone can't completely document anything. For example, a good API might specify that "the sender must be non-null, and must be valid per RFC blah." There's no way to test that inclusively, to check all possible inputs. You can't use the test cases to deduce "we must meet RFC blah." You might suspect it, but you'd be risking undefined behavior it you stray from input that doesn't exactly match the test cases. And before anyone objects "the API docs can be incomplete too," well, that true. But the point is that a written API has vastly more descriptive power than a set of test cases. (The same applies to "self-documenting code". Bah humbug.) There's also the objection "but you can't guarentee cases you don't test!" Also true. That's reality. _You can never test all your intended behavior._ You pick your test cases to do the best you can, and change your cases as problems pop up. The other thing I would shy away from is including throwaway tests in the framework. Throwaways are a thing, developers use them all the time, but don't make them unwanted stepchildren--poorly (incompletely?) designed, slapped together, limited scope, confusing for another developer (including time-traveling self) to wade through and decide whether this is a real failure or just bogus test. They're tech debt. Less frequently used tests are another matter. For example, release-engineering tests that only get run against release candidates. But these should be just as much real, set in stone, as any other deliverable. Which I guess is a third viewpoint nuance difference. I treat tests as being part of the package just as much as any other deliverable. They morph and shift as APIs change, or dependencies mutate, or bugs are found. They aren't something that can be put to the side and left to vegetate. |
Which means the RFC claim is false and should not be asserted in the first place. The API may incidentally accept valid RFC input, but there is no way to know that it does for sure for all inputs. You might suspect it conforms to the RFC, but to claim that it does with certainty is incorrect. Only what is documented in the tests is known to be true.
Everything else is undefined behaviour. Even if you do happen to conform to an RFC in one version, without testing to verify that continues to hold true, it probably won’t.
This is exactly why unit tests are the expected documentation by users. It prevents you, the author, from make spurious claims. If you try, the computer will catch you in your lies.
> The other thing I would shy away from is including throwaway tests in the framework.
What does that mean? I suspect you are thinking of something completely different as this doesn't quite make sense with respect to what I said. It probably makes sense in another context, and if I have inferred that context correctly, I'd agree... But, again, untreated to our discussion.