[liquidise]

Commenting on TDD stories here is historically a bad practice, but i'll add my input here.

I have never let my teams go full TDD. The reason is that in all my experience, TDD sacrifices a lot of velocity for the sake of automated tests. When i hear about the reduction in total bugs injected, it is a "duh" moment. The fastest way to make a team inject 30% fewer bugs is to have them write 30% less code. That isn't snarky, it's true.

Automated testing is one of the many tools available to software engineers. And it is a valuable one. Unfortunately, TDD is too much of a good thing. It relies so heavily on automated testing that it ventures far into the realm of diminishing returns.

Once, in an argument about TDD, i said it was akin to having someone build a shed. But upon checking in on them, you saw they were using a hammer to smash screws into boards. When you ask them what they are doing, they tell you it is Hammer Drive Construction. It is perhaps overly harsh, but it reinforces the point: tools have a place. Automated tests really shine on mission critical logic that does not get rewritten often. Use it where it makes sense. I wouldn't recommend using it ubiquitously.

Then again, i also recommend having fun coding. So i suppose the actual message here is: do what makes you successful, not what comments or studies say.

[crdoconnor]

I find that automated tests shine pretty much all of the time, provided they're relatively cheap to build, cheap to maintain and not buggy.

Where they fall down is when they're more expensive to build than the code under test and they produce false positives/negatives.

[YZF]

I agree. The problem is that almost inevitably they end up getting more and more expensive to build and maintain while at the same time becoming buggy ;) It's a difficult battle to win.

I think you need self discipline to keep limiting yourself to an ever evolving subset that is the optimal ROI. This means over time removing tests that don't add as much value any more. Rewriting some other tests. etc. Human nature though is that these keep growing endlessly and become hard to manage just like any other part of software.

[spronkey]

Hmm, I think this arises from a mindset that doesn't treat tests as part of the code that should be maintained and refactored.

You always have to be removing and refactoring tests if you are changing your production code. Changed a sorting algorithm? Good, go and have a look at your tests to see whether there's anything that doesn't need to be tested anymore, or edge cases that need to be tested now that the algorithm has changed.

Red -> Green -> Refactor

[marcosdumay]

Unit tests tend to get more expensive with time (but not always do). Other kinds of test behave differently.

[pooktrain]

You say "do what makes you successful" after "I have never let my teams go full TDD".

If someone on your team is most successful with TDD, do you still not allow it?

re: writing 30% less code - I've found TDD can reduce my percentage of lines of code, as you suggested. Adherence to the "refactoring" part encourages that you reduce duplication, which in my experience has been easier to do with good test coverage.

[liquidise]

You are correct. My use of the singular "you" was more directed toward people in their own projects. For the purposes of a team at a company, you can think of it as a collective "us". We do not use TDD.

I would say that the most successful teams i have been a part of focus not on automated testing but instead on other collective practices: informal code reviews, diff analysis of every commit, group discussion of database changes and collective manual testing of other's code. Many people point to the refactoring (or initial code organization) as a benefit of TDD. I find these other practices tend to inspire a more collective ownership of the system. Additionally, and more importantly, they spur a lot of conversation around how and why to organize code certain ways. These learning opportunities are probably the most valuable among young and growing teams.

[pooktrain]

Sure, these practices can help build a healthy engineering culture, and I agree with them all, especially collective manual testing.

How do you keep people from doing TDD though? How do you even know they are doing it?

[gozur88]

That's been my experience. I would add that TDD is the antithesis of "agile", since any changes you make to your product will require changes to the tests. Sometimes large changes.

[flukus]

That depends on the change and the tests. Ideally the changes to individual tests should be zero (for irrelevant tests) and changes to the test fixture should be minimal.

Having said that, unit tests in the wild have a tendency to be abominably written, so I'm not surprised a lot of people get frustrated changes the tests.

[alphanumeric0]

In my experience, the large changes to my tests were a result of having to make large changes in product behavior.

[unclebobmartin]

Only if you design your tests that way. Tests are just software. If a change to one part of a software system requires massive changes to another part of that same system; then the system is poorly designed. Indeed, that may be the very definition of poor design.

So if a change to your production code causes large changes to your test code, then one, or the other, or both are poorly designed. You have neglected the design. You have allowed couplings to proliferate.

[keithnz]

so what's your approach to ensuring the software you deploy is correct?

[GrinningFool]

How does TDD prove its correctness? TDD suffers from the same limitations as the code - it generally only covers what you could think of.

It's a powerful tool, but I think any belief that sufficient test coverage (in most common cases) actively proves correctness is misguided. In the general case, even full test coverage proves only that you've tested for the conditions you expect - but does nothing to verify the correctness of behavior in conditions you didn't expect.[1]

To me the benefits of TDD are three-fold:

1. It makes you think of what you're building in more detail before you build it.

2. The methodology puts heavy emphasis on short test-code cycles.

3. (Applies to any methodology that emphasizes coverage) You end up with an acceptable-to-great regression suite[1], and anecdotally it seems people do a better job of at least ensuring tests exist when required to by the methodology.

All of these things are equally possible without TDD. Short iterative cycles and additional forethought are perfectly possible without TDD, but they do require more discipline - it is harder to remember to stop after completing a small set of changes without a forcing mechanism.

[1] rust is a possible exception here, still wrapping my head around it.

[2] the value of this regression suite varies greatly from project to project. A hint that tests are of low value /potentially high cost can be seen when you're finding that minor internal changes either break a large number of tests or reduce coverage to noticeable degree. Particularly in absence of functional changes.

[keithnz]

Correctness is by definition, you say what your code should do in various situations. You can then automatically verify that is the case ( using whatever method ). You need to capture "correctness" in some form. For things you can't think of, and then learn about, you add that stuff to your definition of correctness. I'm not arguing for TDD ( or against ), I'm asking if they don't use TDD, what do they do to capture correctness? I'm interested to know. TDD certainly doesn't try to cover correctness at all levels of software deployment, but it does try to capture fine grained correctness.

Sometimes correctness isn't that valuable compared to other criteria as faults can be quickly corrected and have minimal impact. But I think you need clarity about the tradeoffs you make.

EDIT: in some situations things can be corrected quickly.

[k_sze]

"How does TDD prove its correctness? TDD suffers from the same limitations as the code - it generally only covers what you could think of."

Another limitation is that because the tests are (usually?) written by humans, the tests could also be wrong.

So you think your application works, but it turns out that both your application code and your test code were wrong and you didn't catch a bug at all.

[GrinningFool]

That's the biggest drawback of any kind of tests - there is always the risk of testing the wrong thing, or not testing enough of the possible right things. and you're absolutely right - because the test is also code, it also can and often will have bugs.

When you write tests for what you're about to code, or what you just finished coding, it's a challenge to write a test that is not flawed in the same way the code is - because you don't know the flaw is there in order to test for it.

By thinking things out first you can decrease the number of these (and enforcing that discipline is a major plus for TDD ) but for typical non-trivial application without a lot of control over its inputs, it's close to impossible to do.

Tests are often reactive as well - because the changes that require them are reactive (bugs, new reqs, arch changes, etc). That doesn't detract from the value of them, but it's a limitation that explains well why even 100% coverage never stops the bugs from showing up.

I think I'd be happier with TDD if fewer people presented it as if it solved all the problems. TDD is a powerful tool, but tools are only as omniscient as the people who use them.

[cjwoodall]

Testing validates that the program is correct according to some base of assumptions. You run into this a whole lot in embedded systems, where mocking hardware is difficult. You can't feasibly unit test against real hardware (most of the time), so instead you unit test against your assumption of what the hardware does and verify you respond according to requirements.

Has the benefit of proving correctness of your assumptions, which makes it easier to debug once you insert it in system and things inevitably are not 100% right. It gives you a way to reason about what your code does, what might be different, and then allows you to revise your assumptions and get your new solution in place and tested without the often long wait times to do manual testing on deeply embedded hardware.

Sometimes traditional TDD is the answer, sometimes simulations are the answer, and sometimes you need to just get out of your chair and test it out. It is a tool!

[YZF]

As someone I know likes to say, write better code?

It's a given that software needs to be tested. The processes around that are a classical "it depends" question.

Most likely any software you deploy will never be "correct" (whatever that means). The quality of that software depends on many variables and it's up to you to try and tweak them while optimizing for things like cost, time etc. Whether it's worthwhile to write the tests ahead of time or after the fact or to do them manually or automatically or any other permutation is just not a question that can be answered in a way that applies to all situations.

[gozur88]

>It's a given that software needs to be tested. The processes around that are a classical "it depends" question.

Yes. What you need to do to ensure the correctness of flight software for a jet fighter is entirely different than what you need to do to ensure the correctness of an internal tool at your company that automates a task for which you already have a manual process.

[st3v3r]

That's like saying, if you're poor, you should buy more money.

[pka]

Having a good type system would be a good start :)

[koder2016]

Don't upset the scripters ;).

[keithnz]

I agree :)

[taneq]

You can use automated unit and integration testing without doing "TDD". The overhead isn't even particularly high; you have to test any piece of code you write anyway, so you might as well put in a tiny bit of extra effort and have unit testing.

[GFK_of_xmaspast]

"TDD" is not the same thing as "having unit and integration tests".

[bunderbunder]

I think that this all goes straight back to the old "mockist TDD vs classical TDD" debate.

[convolvatron]

could you elaborate? i think this is my ignorance. The one shop that I worked in that insisted on 'mocks' meant that i wrote some code, then ran that code on some inputs, recorded the outputs, and then wrote a harness which validated that those inputs matched the outputs.

which meant that changes to the code might result in a failed mock, but didn't say anything about coverage or correctness. i can't imagine a more useless testing strategy.

is that what mockist TDD is commonly understood to be?

[bunderbunder]

Yeah, that is.

In what's often termed the "classic" approach, you instead lean toward writing more coarse-grained tests, and you don't shy away from integration tests. You don't avoid mocks, but you tend to prefer saving them for situations where it really is hard to force a collaborator to behave in a certain way. (I also try to stay on guard for the possibility that those situations are code smells indicating that your implementation is getting to be too complex and is due for a refactor.)

IMO, the main argument against classical TDD is that you tend to get a suite of tests that runs more slowly and has more dependencies on external resources such as the database.

IMO, the main arguments against mockist TDD are that you end up with test-induced design damage, and a brittle suite of tests that makes your codebase resistant to refactoring.