[jph]

> Some might claim unit tests will solve this

Yes. Tests will solve this. Your point is perfect for tests.

If another experienced coder cannot comprehend from the tests why something is wrong, then improve the tests. Use any mix of literate programming, semantic names, domain driven design, test doubles, custom matchers, dependency injections, and the like.

If you can point to a specific example of your statement, i.e. a complex method that you feel can be explained in documentation yet not in tests, I'm happy to take a crack at writing the tests.

[Chaebixi]

How do you express "X is a dead end; we tried it and it didn't work because Y, so this is Z" as a unit test? The strength of prose is that it can be used express concepts with an efficiency and fluency that syntactically-correct cannot do.

Sometimes you just have to pick the right tool for the job, and sometimes that tool is prose. I think if you get too stuck on using one tool (e.g. unit tests), you sometimes get to the point where you start thinking that anything that can't be done with that tool isn't worth doing, which is also wrong.

[harlanji]

Literate commit messages. The tool TRAC did a great job of surfacing project activity into timelines and exposing views like that. It's possible with GH but I'm usually the only one on projects to write commit messages that aren't dismissive like "words" or "fdsafdasfas"... soooo.... Release Notes are the best I can do for now.

Bigger still, is what happens when a project spills beyond a single repo, but not even Google is that big :) :). Apache projects are good models for that kind of documentation IMO, even though the pages have ugly css.

[the_af]

> Literate commit messages.

That's a form of written-in-prose documentation like the OP was arguing for, and unlike writing unit tests. Documentation doesn't have to be in a separate file.

[msluyter]

Here I'd distinguish between system/integration and unit tests. Unit tests as a whole tend to amount to a mirror of the code base. If a given function f returns '17' and a test validates that fact, all we've done is double check our work -- which has some value, but doesn't protect against the case in which f is _supposed_ to return 18 and both the code and the test are wrong.

OTOH, system tests provide a realm where external implicit/explicit requirements may actually be validated. Perhaps.

[fao_]

IMO tests should never do this unless the function's role is obvious. This prevents internal restructuring (Which is bad!). What you should instead prefer is end-to-end testing.

So for a lexer, you'd create a dummy program with the output of each token on a newline, and then test that the tokenizer's output matches what you expect from the input. But you shouldn't test whether the functions themselves are correct. The tests cases should be designed to throw up any bugs in any of the internal functions, and the system should catch it as defined.

Otherwise you end up documenting what the system currently is, rather than that the goal of the system is met.

[alien_at_work]

Completely disagree here. If you test this way you'll be sure to have a correct implementation for this dummy program. Further, if the test fails you now have to try and figure out what component caused the failure.

A unit test should test a specific unit; commonly a class. You should have unit tests for the interface of that unit (never private or even protected methods). This absolutely does not prevent internal restructuring but it drive you toward seeing your classes as individual service providers with an interface.

[fao_]

> If you test this way you'll be sure to have a correct implementation for this dummy program.

That's the point. If the dummy program fails then your implementation is bad.

> Further, if the test fails you now have to try and figure out what component caused the failure.

Have you heard of logging?

> A unit test should test a specific unit; commonly a class. You should have unit tests for the interface of that unit (never private or even protected methods).

This is almost exactly what I suggested. I stated 'dummy programs' for individual tests because it's more modular and closer to actual usage than mocking. If your language has an equivalent, use that.

[alien_at_work]

>Have you heard of logging?

So your solution is to dig through log files to find the problem instead of read the unit test name(s) that demonstrate the failure?

>This is almost exactly what I suggested.

Well, it wasn't clear from your wording. It sounded like you were advocating integration testing instead of unit testing (which is a thing that people commonly do, so that's why I reacted to it).

[virgilp]

> a complex method that you feel can be explained in documentation yet not in tests, I'm happy to take a crack at writing the tests.

// This implementation is unnatural but it is needed in order to mitigate a hardware bug on TI Sitara AM437x, see http://some.url

(that's an obvious one; but there are plenty of other cases where documentation is easier than a test)

[jph]

Turns out this is a great example of why tests are better than documentation.

One way to implement this is by using two methods: one normal and one with the hardware mitigation code, with a dispatch that chooses between them.

This separation of concerns ensures that your normal code runs normally on normal hardware, and your specialized code runs in a specialized way on the buggy hardware.

This separation also makes it much clearer/easier to end-of-life the mitigation code when it's time.

[virgilp]

A. You took it too literally ( maybe it's not a hw bug on some exotic processor, maybe it's a mitigation for a security vulnerability that affects a broad set of OS distros).

B. Your solution is not obviously better, it's a different trade-off. And it's not immediately apparent to me how exactly you would write the test for the affected hardware, in the first place. What if the hardware bug is extremely hard to reproduce?

[jph]

Fair points.

A. When I encounter areas that need specialized workarounds (such as a mitigation for a security vuln) then I advocate using two methods: one of the normal condition and one for the specialized workaround. Same reasons as above, i.e. separation of concerns, easier/clearer normal path, easier to end of life the workaround.

B. In my experience tests are better than documentation for any kind of mitigation code for an external dependency bug, because these are unexpected cases, and also the mitigation code is temporary until the external bug is fixed.

Imagine this way: if a team uses just documentation, not tests, then what's your ideal for the team to track when the external bug is fixed, and also phasing out the mitigation code?

[virgilp]

> Imagine this way: if a team uses just documentation, not tests, then what's your ideal for the team to track when the external bug is fixed, and also phasing out the mitigation code?

- not all external bugs are fixed, some are permanent (e.g. the hardware issues).

- tests can tell you when something is wrong, but not when something is working. I'm not really sure how a test could tell you that "external bug is now fixed" - and even in cases where that _might_ work, I'm not sure it's a good idea to use a test for that.

[kbenson]

Do you think this still holds true if you name all your tests in the format test1, test2 ... testN? If not, then you're in the realm of documentation, not tests, and the descriptive names (which is a form of metadata, just as comments are) of the tests are what is communicating these special cases, and not the test content itself.

Combining the two is good, but let's not act like the tests themselves immediately solve the problem.

[jph]

My opinion is that test names, function names, variable names, constant names, high level languages, literate programming, and well written commit messages, all help code to be understandable; I'm fully in favor of all using all these in source code and also in commit messages.

My experience is that documentation is generally a shorthand word that means non-runnable files that do not automatically get compared to the application source code as it changes.

Of course there are some kinds of blurred lines among tests and documentation, such as Cucumber, Rational, UML, etc.; but that's not what the parent comment was talking about when they described the function with a naive/buggy implementation vs. an enhanced implementation that handles a subtle case.

> but let's not act like the tests themselves immediately solve the problem

I'm saying that yes, the tests do immediately solve the problem in the parent comment's question: a test for the "subtle" case in the parent comment immediately solves the problem of "how do we ensure that a future programmer doesn't write a simplified naive implementation that fails on this subtle case?"

[kbenson]

That wasn't the parent's case though. The parent's case was an odd implementation that provided correct enough approximations of the answer in a much more performance way. Neither are incorrect, but the "simplified" version would be a step backwards.

As noted, the comments are for the why, which tests don't tell you without some additional information.

[maxxxxx]

Sometimes I wonder if we should mark tests as "documentation", "internal" or similar. When I get a pile of unit tests I find it really hard to tell which ones are for the overall system vs. testing just a detail that may change.

[bhntr3]

  import cPickle
  def transform(data):
    with open('my.pkl', 'r') as f:
      model = cPickle.load(f)
      return model.predict(data)

[Retra]

If you change code and break tests doing something that is known to be wrong, you are wasting your time and everyone else's.

[squeaky-clean]

Old and somewhat contrived example, but the first thing to pop into my head is the famous fast inverse square root function.

    float FastInvSqrt(float x) {
      float xhalf = 0.5f * x;
      int i = *(int*)&x;         // evil floating point bit level hacking
      i = 0x5f3759df - (i >> 1);  // what the fuck?
      x = *(float*)&i;
      x = x*(1.5f-(xhalf*x*x));
      return x;
    }

I can't think of a way to write a test that sufficiently explains "gets within a certain error margin of the correct answer yet is much much faster than the naive way."

The only way to test an expected input/output pair is to run the input through that function. If you test that, you're just testing that the function never changes. What if the magic number changed several times during development, do you recalculate all the tests?

You could create the tests to be within a certain tolerance of the number. Well how do you stop a programmer from replacing it with

    return 1.0/sqrt(x);

And then complaining when the game now runs at less than 1 frame per second?

Here's a commented version of the same function from betterexplained.com.

    float InvSqrt(float x){
        float xhalf = 0.5f * x;
        int i = *(int*)&x;            // store floating-point bits in integer
        i = 0x5f3759df - (i >> 1);    // initial guess for Newton's method
        x = *(float*)&i;              // convert new bits into float
        x = x*(1.5f - xhalf*x*x);     // One round of Newton's method
        return x;
    }

It's still very magic looking to me, but now I get vaguely that it's based on Newton's method and what each line is doing if I needed to modify them.

I actually just found this article [0] where someone is trying to find the original author of that function, and no one on the Quake 3 team can remember who wrote it, or why it was slightly different than other versions of the FastInvSqrt they had written.

> which actually is doing a floating point computation in integer - it took a long time to figure out how and why this works, and I can't remember the details anymore

This made me chuckle. The person eventually tracked down as closest to having written the original thing had to rederive how the function works the first time, and can't remember exactly how it works now.

I think the answer is both tests and documentation. Sometimes you do need both. Sometimes you don't, but the person after you will.

[0] https://www.beyond3d.com/content/articles/8/

[jph]

For example, here's one way to write a test that sufficiently explains "gets within a certain error margin of the correct answer yet is much much faster than the naive way".

Using Ruby and its built-in minitest gem:

1. Write a test that does minitest assert_in_epsilon(x,y,e)

2. Write a minitest benchmark test that compares the speed of the fast function with the speed of the naive function.

Notice the big advantage for long term projects: if the hack ever ceases to work then you'll know immediately. This actually happens in practice, such as math hacks that use 32-bit bit shifts that started failing when chip architecture got wider.

> no one on the Quake 3 team can remember who wrote it

Exactly. We have the code file, but not any documentation separate from the code, such as notes, plans, attempts, reasoning, etc.

[squeaky-clean]

> Exactly. We have the code file, but not any documentation separate from the code, such as notes, plans, attempts, reasoning, etc.

I agree with you that it can be tested. But it doesn't explain anything about why it works, or what methods the author tried that didn't work as well. If you ever had to make it faster, or make it work better on different hardware, you'd be starting from scratch again.

[jph]

> If you ever had to make it faster, or make it work better on different hardware, you'd be starting from scratch again.

Optimizations like these a great area for tests because the test files can keep all the various implementations and can benchmark them as you like.

This enables the tests to prove that the a new implementation is indeed optimal over all previous implementations, and continues to be optimal even when there are changes in external dependencies such as hardware, libraries, etc.

> But it doesn't explain anything about why it works

IMHO it does, for all the areas expressed in the original link and the parent comment.

Here are the original link examples:

1. "What [TDD] doesn’t do is get you to separate the code’s goals from its implementation details."

IMHO first write the code's goals as tests, then write the method implementations. The tests may need new kinds of instrumentation, benchmarking, test doubles, etc.

2. "[D]oes the description of what Module 3 does need to be written in terms of the description of what Module 1 does? The answer is: it depends."

IMHO write modules with separation of concerns, and with clear APIs that use API tests. If you want to integrate modules, then write integration tests.

3. "Quick! What does this line of code accomplish? return x >= 65;"

IMHO write code more akin to this pseudocode:

    return age >= US_RETIREMENT_AGE

    return letter >= ASCII_A

[kazagistar]

Write a property based test, ie generate a bunch of random inputs, then assert that all of them are within some (loose) margin of error.

[Chaebixi]

> Write a property based test, ie generate a bunch of random inputs, then assert that all of them are within some (loose) margin of error.

So someone comes along later, look at your test, and wonders: why did you go through all that trouble? You can definitely write tests for a lot of that stuff, but they still don't fluently communicate the why of your choices.

[squeaky-clean]

This doesn't satisfy the time constraint though.

    return 1.0f / sqrt(x)

Passes a property based test but now your game doesn't actually run because it's much too slow of an operation on hardware at that time.

You can also test execution time too, but that's finicky and doesn't help explain how to fix it if you break that test (if there's no accompanying documentation).

[tome]

But at this point all you're saying is "I can't think of a way of testing performance".

[reacweb]

Performance can be tested in a unit test. You just need to measure the time needed to compute the function on a given set of numbers, then measure the time needed to compute 1.0f / sqrt(x) on the same set of numbers. The test succeed if your function is 10x faster. In future, the test may fail because sqrt has improved and this trick is no more needed.

[squeaky-clean]

No I'm saying a test for performance doesn't accurately describe the reasoning behind it without accompanying documentation.

Speed is the entire purpose of this method, not the numerical accuracy.