[couchand]

I think you're spot on here - TDD is great as long as you're not too obstinate about it. It's a trade off, just like every interesting problem.

One point I'd like to draw out. If you don't have a clue how you want to build things, TDD isn't going to help.

This is exactly right. If you find yourself completely unable to articulate a test for something, you probably don't really know what it is you're trying to build. I think that's the greatest benefit to TDD: it forces you to stop typing and think.

[bphogan]

Exactly. This is the whole purpose behind the "spike" - make a branch, write a crap implementation of some code to help understand the problem, put it aside. Then go write the production version TDD style. Once you understand the problem, you can use TDD to create a good design to solve that problem.

Sounds crazy, but this is how I do everything I don't understand. And my second implementation is usually better than my first.

[couchand]

Or, in the words of Fred Brooks, build one to throw away. I'm always amazed at how prescient he was.

Unfortunately I find all too often that spike project finds its way into production for one reason or another. Now I only spike in Befunge.

[Chris_Newton]

If you find yourself completely unable to articulate a test for something, you probably don't really know what it is you're trying to build.

I don’t buy this argument. How would you write tests to drive the development of a graphics demo, say rendering a Mandelbrot set? Or a tool to convert audio data from one format to another? Or any other kind of software where the output doesn’t consist of readily verifiable, discrete data points?

[couchand]

Are you asking about unit tests or acceptance tests?

The problems you describe are very high level, but we could design an acceptance testing scheme for them. For the Mandelbrot set it might involve comparison to a reference rendering, for the audio tool a reference recording. In both cases you'd allow a delta relevant to the application, and probably also benchmark for acceptable performance.

But my point was more aimed at unit testing. When you set out to write a function you should know something about that function before starting. If you know enough to write the function signature, you can first write a failing test. If you can write a bit of code in that function, you can write a bit expecting the behavior of that code.

[Chris_Newton]

Are you asking about unit tests or acceptance tests?

I suppose what I’m really asking is how you would go from not having software to having software that does those things, using TDD. I think in practice its fail-pass-refactor cycle is normally applied at the level of unit tests, but in any case, how would using TDD help to drive a good design, to ensure testability, or otherwise, in that kind of situation?

(I’m asking this rhetorically. I don’t think TDD is a very helpful process in this context. I’m just trying to demonstrate this with practical examples rather than bluntly stating it without any supporting argument.)

[colomon]

I think I mostly agree with your larger point, but I'm not in love with your examples. The Mandelbrot set does consist of readily verifiable discrete data points, after all. I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

A great example for your point (which might have been what you were getting at with the audio thing) is a test for creating files in a lossy audio format. The acid test is if it sounds right to a human being with good ears, I've got no clue how you would write a pure computer test for that.

In my own work, a great example is finding a NURBS approximation of the intersection of two surfaces. There are an infinite number of correct answers for a given pair of surfaces, and testing that the curve you've generated fits the surfaces is a distressingly hard problem.

[Chris_Newton]

The Mandelbrot set does consist of readily verifiable discrete data points, after all.

Indeed it does, but in order to verify them I see only two options.

One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all.

The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place.

This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose.

I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

I’m genuinely curious about how you’d see that going.

[colomon]

I kind of wish I had the time to actually do it right now and see how it works. But here's how I imagine it going:

1) Establish tests for the is-in-set function. You're absolutely right that the most obvious way to do this meaningfully is to reimplement the function. A better approach would be to find some way to leverage an existing "known good" implementation for the test. Maybe a graphics file of the Mandelbrot set we can test against?

2) Establish tests that given an arbitrary (and quick!) is-in-set function, we write out the correct chunk of graphics (file?) for it.

3) Profit.

Observations: 1) I absolutely would NOT do the "write a test; write just enough code to pass that test; write another test..." thing for this. My strong inclination would be to write a fairly complete set of tests for is-in-set, and then focus on making that function work.

2) There's really no significant design going on here. I'd be using the exact same overall design I used for my first Mandelbrot program, back in the early 90s. (And of course, that design is dead obvious.)

In my mind, the world of software breaks down something like this: 1) Exhaustive tests are easy to write. 2) Tests are easy to write. 3) Tests are a pain to write. 4) Tests are incredibly hard to write. 5) Tests are impossible to write.

I think it's pretty telling that when TDD people talk about tests that are hard to write, they mean easy tests in hard to get at areas of your code. I've never heard one discuss what to do if the actual computations are hard to verify (ie 4 & 5 above) and when I've brought it up to them the typical response is "Wow, guess it sucks to be you."

[couchand]

Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.

I think you get to the nub of it here. TDD lets you develop a spec that is consistent with requirements (the subset so far implemented) and the code at all times.

Writing a comprehensive suite of tests before any production code is like writing a complete spec without any clue as to its applicability. Writing tests afterward would be analogous to writing a spec for an already shipped product.

Tests work both ways in TDD: you are checking both that the code behaves as intended and that your expected behavior is reasonable. If it were only about the former it wouldn't be very valuable.