[the_af]

A Cold Shower for (early) testing of software, maybe:

There used to be an often cited paper by Boehm about the cost of catching bugs early vs late on production, usually mentioned by advocates of testing early, where the quoted conclusion was something like "studies show it's 10 times more costly to catch bugs late on production" or something like that. This is a very well known study, I'm likely misquoting it (the irony!) and readers here are probably familiar with it or its related mantra of early testing.

I haven't read the paper itself (I should!), but later someone claimed that a- Boehm doesn't state what people quoting him say he said, b- the relevant studies had serious methodological problems that call into question the conclusion he did say, c- there are plenty of examples where fixing bugs late on production wasn't particularly costly.

edit: I'm not arguing testing isn't necessary, in case that upset someone reading this post. I'm not really arguing anything, except that the study by Boehm that most people quote was called into question (and was probably misquoted to begin with). This doesn't prove/disprove anything, except maybe hinting at a possible Cold Shower. It does show that we as a field have a serious problem in software engineering with backing up claims with well designed studies and strong evidence, but this shouldn't come as a surprise to anyone reading this.

[hwayne]

Laurent Bossavit tears the Boehm paper apart in his book "Leprechauns of Software Engineering"[1]. It's a good read for anyone interested in the empirical side of software research.

[1]: https://leanpub.com/leprechauns

[the_af]

Thanks! I think it might have been an extract from Bossavit's book what I'm struggling to remember now. The title definitely rings a bell.