[akkartik]

Perhaps it suggests our way of testing needs to change? A while back I wrote a post describing some experiences using white-box rather than black-box testing: http://web.archive.org/web/20140404001537/http://akkartik.na... [1]. Rather than call a function with some inputs and check the output, I'd call a function and check the log it emitted. The advantage I discovered was that it let me write fine-grained unit tests without having to make lots of different fine-grained function calls in my tests (they could all call the same top-level function), making the code easier to radically refactor. No need to change a bunch of tests every time I modify a function's signature.

This approach of raising the bar for introducing functions might do well with my "trace tests". I'm going to try it.

[1] Sorry, I've temporarily turned off my site while we wait for clarity on shellsock.

[thisGuysAccount]

Something to consider, and this is only coming off the top of my head, is introducing test points that hook into a singleton.

You're getting more coupling to a codebase-wide object then, which goes against some principles, but it allows testing by doing things like

function awesomeStuff(almostAwesome) {

  MoreAwesomeT f1(somethingAlmostAwesome) {
    TestSingleton.emit(somethingAlmostAwesome);
    var thing = makeMoreAwesome(somethingAlmostAwesome) 
      // makeMoreAwesome is actually 13 lines of code,
      // not a single function
    TestSingleton.emit(thing);
    return thing;
  };

  AwesomeResult f2(almostAwesomeThing) {
    TestSingleton.emit(almostAwesomeThing);
    var at = makeAwesome(awesomeThing); 
      // this is another 8 lines of code. 
      // It takes 21 lines of code to make somethin 
      // almostAwesome into something Awesome, 
      //and another 4 lines to test it.
      // then some tests in a testing framework
      // to verify that the emissions are what we expect.
    TestSingleton.emit(at);
    return at;
  }

  return f2(f1(almostAwesome));

}

in production, you could drop testsingleton. In dev, have it test everything as a unit test. In QA, have it log everything. Everything outside of TestSingleton could be mocked and stubbed in the same way, providing control over the boundaries of the unit in the same way we're using now.

[robrenaud]

How brittle are those tests though?

I've had to change an implementation that was tested with the moral equivalent to log statements, and it was pretty miserable. The tests were strongly tied to implementation details. When I preserved the real semantics of the function as far as the outside system cared, the tests broke and it was hard to understand why. Obviously when you break a test you really need to be sure that the test was kind of wrong and this was pretty burdensome.

[akkartik]

I tried to address that in the post, but it's easy to miss and not very clear:

"..trace tests should verify domain-specific knowledge rather than implementation details.."

--

More generally, I would argue that there's always a tension in designing tests, you have to make them brittle to something. When we write lots of unit tests they're brittle to the precise function boundaries we happen to decompose the program into. As a result we tend to not move the boundaries around too much once our programs are written, rationalizing that they're not implementation details. My goal was explicitly to make it easy to reorganize the code, because in my experience no large codebase has ever gotten the boundaries right on the first try.