[couchand]

Are you asking about unit tests or acceptance tests?

The problems you describe are very high level, but we could design an acceptance testing scheme for them. For the Mandelbrot set it might involve comparison to a reference rendering, for the audio tool a reference recording. In both cases you'd allow a delta relevant to the application, and probably also benchmark for acceptable performance.

But my point was more aimed at unit testing. When you set out to write a function you should know something about that function before starting. If you know enough to write the function signature, you can first write a failing test. If you can write a bit of code in that function, you can write a bit expecting the behavior of that code.

[Chris_Newton]

Are you asking about unit tests or acceptance tests?

I suppose what I’m really asking is how you would go from not having software to having software that does those things, using TDD. I think in practice its fail-pass-refactor cycle is normally applied at the level of unit tests, but in any case, how would using TDD help to drive a good design, to ensure testability, or otherwise, in that kind of situation?

(I’m asking this rhetorically. I don’t think TDD is a very helpful process in this context. I’m just trying to demonstrate this with practical examples rather than bluntly stating it without any supporting argument.)

[colomon]

I think I mostly agree with your larger point, but I'm not in love with your examples. The Mandelbrot set does consist of readily verifiable discrete data points, after all. I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

A great example for your point (which might have been what you were getting at with the audio thing) is a test for creating files in a lossy audio format. The acid test is if it sounds right to a human being with good ears, I've got no clue how you would write a pure computer test for that.

In my own work, a great example is finding a NURBS approximation of the intersection of two surfaces. There are an infinite number of correct answers for a given pair of surfaces, and testing that the curve you've generated fits the surfaces is a distressingly hard problem.

[Chris_Newton]

The Mandelbrot set does consist of readily verifiable discrete data points, after all.

Indeed it does, but in order to verify them I see only two options.

One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all.

The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place.

This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose.

I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

I’m genuinely curious about how you’d see that going.

[colomon]

I kind of wish I had the time to actually do it right now and see how it works. But here's how I imagine it going:

1) Establish tests for the is-in-set function. You're absolutely right that the most obvious way to do this meaningfully is to reimplement the function. A better approach would be to find some way to leverage an existing "known good" implementation for the test. Maybe a graphics file of the Mandelbrot set we can test against?

2) Establish tests that given an arbitrary (and quick!) is-in-set function, we write out the correct chunk of graphics (file?) for it.

3) Profit.

Observations: 1) I absolutely would NOT do the "write a test; write just enough code to pass that test; write another test..." thing for this. My strong inclination would be to write a fairly complete set of tests for is-in-set, and then focus on making that function work.

2) There's really no significant design going on here. I'd be using the exact same overall design I used for my first Mandelbrot program, back in the early 90s. (And of course, that design is dead obvious.)

In my mind, the world of software breaks down something like this: 1) Exhaustive tests are easy to write. 2) Tests are easy to write. 3) Tests are a pain to write. 4) Tests are incredibly hard to write. 5) Tests are impossible to write.

I think it's pretty telling that when TDD people talk about tests that are hard to write, they mean easy tests in hard to get at areas of your code. I've never heard one discuss what to do if the actual computations are hard to verify (ie 4 & 5 above) and when I've brought it up to them the typical response is "Wow, guess it sucks to be you."

[Chris_Newton]

1) I absolutely would NOT do the "write a test; write just enough code to pass that test; write another test..." thing for this. My strong inclination would be to write a fairly complete set of tests for is-in-set, and then focus on making that function work.

The latter is what I’d expect most developers who like a test-first approach to do. I don’t see anything wrong with it, either. I just don’t think it’s the same as what TDD advocates are promoting.

I think it's pretty telling that when TDD people talk about tests that are hard to write, they mean easy tests in hard to get at areas of your code. I've never heard one discuss what to do if the actual computations are hard to verify (ie 4 & 5 above) and when I've brought it up to them the typical response is "Wow, guess it sucks to be you."

Indeed. At this point, I’m openly sceptical of TDD advocacy and consider much of it to be somewhere between well-intentioned naïveté and snake oil. There’s nothing wrong with automated unit testing, nor with writing those unit tests before/with the implementation rather than afterwards. Many projects benefit from these techniques. But TDD implies much more than that, and it’s the extra parts — or rather, the idea that the extra parts are universally applicable and superior to other methods — that I tend to challenge.

Thus I object to the original suggestion in this thread, which was that a developer probably doesn’t know what they are doing just because they can’t articulate a test case according to the critic’s preferred rules. I think those rules are inadequate for many of the real world problems that software developers work on.

[colomon]

I almost feel like we should come up with a "How the hell would you test this?" challenge for TDD advocates. At least, my impression is it is mostly naïveté rather than snake oil.

[couchand]

Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.

I think you get to the nub of it here. TDD lets you develop a spec that is consistent with requirements (the subset so far implemented) and the code at all times.

Writing a comprehensive suite of tests before any production code is like writing a complete spec without any clue as to its applicability. Writing tests afterward would be analogous to writing a spec for an already shipped product.

Tests work both ways in TDD: you are checking both that the code behaves as intended and that your expected behavior is reasonable. If it were only about the former it wouldn't be very valuable.

[Chris_Newton]

I think you get to the nub of it here. TDD lets you develop a spec that is consistent with requirements (the subset so far implemented) and the code at all times.

This is another TDD-related argument that I just don’t understand.

A specification might say that the function add returns the sum of its arguments.

A unit test might verify that add(1,1) = 2.

One of these describes the general case. One of them describes a single specific case. Unless your problem space is small enough to enumerate every possible set of inputs and the expected result for each of them, no amount of unit tests can replace a full specification of the required behaviour. Unfortunately, not many real world problems are that convenient.