[pshirali]

I find this article misleading; with a biased PoV. The author says "empirical evidence is missing", yet advocates a testing methodology, banking largely on articles and PoVs of other individuals. Is there any software project the author would like to quote as a case-study? Is there any evidence to prove that the Lean method is better OR, the definition of what is better?

It is OK to be economical in what one would see as a cost of testing a product end-to-end. Call it whatever term you like, but functionally testing a product end-to-end has always followed an economical approach, as all such testing is a factor of time & resources. You'd perhaps be less economical if you were testing life-safety systems, but you could choose to be more economical if you considered your software to be not-that-critical, or, you needn't have to set exceptionally high standards.

I find it extremely odd that unittests are even considered into this equation of 'testing-costs', despite decades of improvements in software development processes. Unittests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unittests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future.

It's a sad state of affairs if one gives code 1st level treatment, but treats unittests, documentation, build scripts and other support infra as something less important; worth economizing. This is really what _matters_ when it comes to overall quality of a software.

One must remember that software is always improved, refactored, expanded, ported or worked on in some way or another. When unittests are missing, then the very boundaries that were meant to dictate the rules of the software don't apply anymore. This leads to human errors causing portions of software to break.

Critical pieces of software that exist today, exist strongly because they were engineered right (Linux kernel for example). Not because their developers followed an _economical_ approach to testing. If an OS disto claimed to perform economical end-to-end testing (of a potential user's most commonly performed paths), would the author of this article want to use that OS over one that has had strong ground-up unittests and testing of interfaces where they matter?

[Brotkrumen]

His claim of lacking empirical evidence shows his lack of research on the subject. There's tons of research that consistently shows comprehensive testing during development makes your product better and cheaper over its lifetime.

Just a quick google and not even the paper I was thinking of: https://softwareengineering.stackexchange.com/questions/6050...

[mdip]

I'd love to see some research around time cost in this area. My personal experience is that writing unit, integration and end-to-end testing ends up either being a wash or reducing the total time it takes to build a product[0].

The rule of thumb I always come back to is "the minute you decide it's time to attach a debugger, be prepared to burn an hour." I used to not mind firing up the debugger -- heck, I used to run my code with a debugger attached every time I ran it. These days, I write tests. I don't aim for 100% code coverage, though I aim for a majority and 100% coverage of critical paths. When I encounter a problem that isn't covered by a test (rare), I write a test centered around the problem. This usually surfaces the bug before I've even ran the actual test -- and once that test is there, any refactoring that affects that code will avoid that bug if the test passes.

This rule has been so effective for me that I've now gone through four different teams, encouraging each to do "the next project" with automated testing. The result has been a commitment to working that way from that point forward[1].

[0] Assuming some baseline quality requirements -- automated build, and a general requirement to publish a mostly functional product. :)

[1] I can't take full credit for the idea -- it was a blog post that convinced me to try it on myself and I advocated that idea after it convinced me.

[chopin]

For time cost, my rule of thumb is that unit tests roughly take the same amount of test code than the code being tested and about the same (mental) effort. Therefore I don't strive for 100% code coverage but I like all critical (or difficult to understand) code being (unit) tested.