[Morendil]

I'm a big fan of TDD, but I quit reading this article the second my eyes came across "IBM System Sciences Institute" and the chart that accompanies it.

This is by now one of the most thoroughly debunked memes in our profession. I wrote a couple chapters about it: http://leanpub.com/leprechauns . I wrote a handy little guide for people to know just how much BS was involved in any one citation: https://plus.google.com/+LaurentBossavit/posts/aNKut1QV8pT

If that wasn't enough, there is now an actual negative research result on the so-called defect cost increase: https://arxiv.org/pdf/1609.04886.pdf

It should be quite clear by now that this type of argument doesn't do TDD any favors, it taints it by association with intellectual dishonesty.

[Hasknewbie]

I can't help but notice that this author, seemingly concerned with correctness, also wrote a post about static typing's benefits being unclear, with his argument there again based on an arbitrary and completely unscientific graph pulled from another Medium blogger.

He seems to base his views on the hearsay from his Javascript monoculture, while ignoring the more relevant literature like Code Complete or (when it comes to TDD) Making Software: What Really Works, and Why We Believe It, both books basing their claims on empirical research. Making Software's conclusion is that TDD can help, but does push complexity upward (i.e. programs end up with simpler components so as to make TDD easier, but with a higher numbers of them and therefore more dependencies between them) so resulting bugs will be harder to track.

But it's hardly surprising this article is poorly researched: it's just rehashing TDD pablum for self-promotion, there is not a single original thought in there.

[mannykannot]

I hung on a bit longer, but lost any hope of seeing something relevant when he launched into an imaginary case study. Later on, we find him mentioning an actual study (of code reviews), only to dismiss its results as being implausible to him.

The author appears to have drunk so deeply of the dogma that he has no idea of how to make an objective case.

[whipoodle]

I thought it was funny he waved off the code review study as well. If the studies don't actually mean anything then why can't we similarly disregard the one that says TDD is good?

[ckastner]

I briefly glanced over the paper, and in references to the 171 projects studied, the following stood out:

> The projects were mostly of short duration and small to medium size. The median project began in 2012 lasted 61 days, produced 4,200 Lines of Code, 49 components (modules or features). Duration ranged from 7 to 1,918 days. Size ranged from minimal (this may represent a short maintenance project) to 88,394. The earliest project was in 2006 and the most recent in 2014.

And from Fig. 11:

> Team size (median): 6

I suspect that a sample of projects involving larger teams, or a larger codebase, would have produced a somewhat more pronounced effect.

Furthermore, it is not clear to me what domains these projects where from (but I admittedly only skimmed the article). In the financial industry, for example, the defect cost increase is certainly more pronounced.

[_Codemonkeyism]

Thanks for that book, one of the very few which are concerned with facts beside the Glass book. Thanks, highly appreciated for years!

[blub]

The Glass book Facts and fallacies of software engineering or another Glass book? :)

[codemac]

Hm, I thought the leading person around the concept of cost increases later in the cycle was actually Capers Jones, and his text about it and cleanroom: The Economics of Software Quality.

Obviously this is NOT about TDD, but about Cleanroom, but it seems like you've done a lot of research in the area.

[blub]

Actually I'd be curious if you've done an in-depth analysis of the available TDD studies. What are the benefits compared to simply writing unit tests that made you a big fan?

I'm aware of Oram/Wilson2010.

[Morendil]

Not in-depth across the whole set; I picked apart one or two before coming to the conclusion that there was something inherently flawed in the whole approach of modeling study of the phenomenon after the protocol for medical intervention. That won't work; you can't do a double-blind study of TDD vs placebo. It's a conceptual tool, and as such requires knowledge and skill.

"Treating" a convenience sample of students with a 2-hour training session and trying to measure "performance" afterwards isn't going to yield much insight into what goes on in the mind of a seasoned programmer who's used TDD exclusively for a while. In my experience, the effects of such practice most definitely carry over to programming where you don't even try to apply TDD.

What convinced me to try TDD was a combination of naivete and desperation. What kept me at it was the undeniable, if entirely anecdotal, feeling of wrestling back control over the code I wrote. It's a more complex set of techniques than just "writing the test first", but it's hard to tease them apart and introduce them into your coding independently of each other.

Part of it is a commitment to evolving a program in very small steps, at every step having a whole program that works. Part of it is a set of instincts for refactoring, so that this program also has just as much "design" (if such a thing could be quantified) as is strictly necessary for the time being. Part of it is a habit of framing the next capability as a tiny experiment, with the test being the experimental protocol, and focusing your mind on just those interactions within the code that are relevant. Part of it is a way of "slow debugging", of not jumping to conclusions when you encounter unexpected behaviour but drilling down into what made the behaviour surprising, and revising your mental model of the program.

Put like that it's clear that to "simply write unit tests", for instance to check a box in a process model, cannot possibly give you the same benefits as practicing the above interrelated set of skills. But it's also clear that it requires more involved guidance than "just write the test first". You can pick up a lot of it on your own, to be sure, just like many people can learn (say) guitar on their own and get all sorts of nuance in technique from focused practice.

But you have to love doing it, and that's not something easily picked up in academic studies.

[blub]

Nicely explained, thank you.