[zuzululu]

TDD sounds great on paper for agentic development but you quickly realize it balloons the token cost. Often I write some feature and then its repurposed or removed, code is refactored moved around as time goes. With TDD I would be taxed heavily and velocity slow to a crawl.

The waterfall approach is better after trying out TDD especially when you have a multi-agent setup. Also I found that in some cases the tests were just superficial hallucinations that never actually tested the components written or there some some context corruption and ultimately triggered a false positive that kicked off a completely unintentional refactoring.

[__mharrison__]

My experience is the opposite. TDD keeps the guardrails on and let's me refactor with confidence.

Crazy times here in the development world. I'm always curious to watch other's best practices.

[dools]

Yeah I specifically tell it not to pre-emptively fix tests that it knows will break as a result of changes its making and instead limit itself only to creating new tests for new changes. I want to see the tests break, then we go through and review each set of breakages versus the mission and assess if they’re regressions or stale assertions. This is a) how I know it’s actually writing meaningful tests b) a very functional and useful form of “code review” versus just trying to catch problems by reading diffs and c) helped me find real problems and regressions.

Almost all the breakages after a big refactor are stale assertions but every time I catch a couple of critical problems that make the entire exercise very worth it.

The whole dev process is so fast compared to writing software manually that I find it absurd that I wouldn’t invest heavily in automated tests.

[__mharrison__]

See my AGENTS.md in nearby comment

[rsalus]

I was a big proponent of encoding TDD red-green-refactor methodology into my agent workflows until recently when I made the same realization after reading this study: https://arxiv.org/pdf/2602.07900

TLDR; it found test-writing volume only weakly correlates with success and that encoding test-writing principles did not move resolution rates but _did_ materially change cost. Encouraging tests cost +19.8% output tokens for 0% gain; discouraging them saved 33–49% input tokens for ≤2.6pp accuracy loss. Separately, imposing the TDD procedure specifically seems like it can backfire: it actually _increased_ regressions from 6.08% to 9.94%.

IMO, where tests clearly help is primarily as an "oracle" applied after generation. It gives the models a signal that enables them to verify and self-correct if necessary.

[zuzululu]

Very interesting paper and it lines up exactly with my observations. The ROI just isn't there writing tests up front and the conclusion in that paper lays it out clearly

    Overall, these findings suggest that agent-written
    tests often behave more like a habitual software-development rou-
    tine than a dependable source of validation in this setting. More
    agent-written tests do not mean more solves; what they more reli-
    ably change is the process footprint—API calls, token usage, and
    interaction patterns. Improving the value of testing for code agents
    may therefore require better oracles and more actionable validation
    signals, rather than simply inducing agents to write more tests.

> IMO, where tests clearly help is primarily as an "oracle" applied after generation

Bingo. I'm not against writing tests it's that the returns are better when its used as verification feedback and as "Oracle" exactly as you put it.

[girvo]

Just chiming in to say that I've seen the exact same that you have. Tests are better used to help validate that was was generated worked after the fact.

That, and even the absolute SOTA models still suck at writing tests.

Which shouldn't be surprising: humans suck at it too most of the time...

[zuzululu]

Absolutely, there's no reason to believe that agents will be more capable of writing tests than any other piece of code. The big pay off is actually verifying the code that was generated.

[esperent]

From that paper:

> This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget?

This is an important question but it's not the one I'm most interested in when requiring agents to follow TDD. My goal is to lock in behavior because it was happening way too frequently that an agent would successfully fix the issue at hand, but break something else that it wasn't supposed to touch.

The tests add another layer and it's why I always separate out red and green worker subagents. The green worker might get trigger happy and go beyond scope/break something but it's not allowed to fudge the tests so I'll know and can clean up and revert.

It's also why I'm not too bothered about perfect red green TDD. I can add the tests later if needed.

[rsalus]

tests are an important signal of course, but the use case you describe doesn't necessarily mean you need to follow TDD. the data suggests that creating the tests after the code is just as or even more effective, and at significantly cheaper input cost.

I've been finding enforcing integrations and behavior structurally (e.g., through codegen/schemagen, e2e tests, etc) more reliable than simply instructing the models to write tests. oftentimes these tests are pretty low quality anyway, and results in its own form of tech debt.

[esperent]

Why do you think creating tests later would be cheaper?

[necovek]

The paper focuses on two things: default behavior and behavior with a prompt to write at least one new test.

In general — just like with humans — I find "just add more tests" to be counter-productive.

Tests make sense in a testable architecture: TDD can encourage one to be implicitly used, but it is a design, architectural choice that should be made explicit (lean to functional code; use direct, explicit dependency injection; ensure test stubs are just variants of the real implementation and fully tested using the same test as the real one...). LLMs should be prompted with this guidance instead for proper value estimation.

[dnautics]

no. red green tdd is great because you'll have tests when your llm breaks something later, or you're doing a massive refactor. i imagine studies are not done on codebases where the complexity gets that high.

tdd has been invaluable for this project (almost entirely llm written, but i review it) https://github.com/ityonemo/clr

[rsalus]

this is not really backed by any empirical evidence. there are simply more efficient means of verifying outputs than TDD.

[pramodbiligiri]

My approach (with LLMs especially) aligns more with what's outlined in "Growing OO Software Guided by Tests" (https://growing-object-oriented-software.com/toc.html). Chapter 4 there says "First, Test a Walking Skeleton", and Chapter 5 has "Start Each Feature with an Acceptance Test". I think it comes down to: get something working end-to-end first in a verifiable way, and then keep refining both the feature and its tests (preferable with TDD).

I've noticed that LLMs tend to generate multiple testcases in one shot (which is not how humans usually go about TDD), and also they don't start with Integration Tests, unless instructed to do so.

[dnautics]

> it balloons the token cost

how!!??

you write a test, which is one extra function. and maybe a paragraph or so per feature ("i made a RED test"... "i made it GREEN"), everything else is the same between normal development and TDD. this is chump change compared to the rest of development, including thinking tokens

[manmal]

> With TDD I would be taxed heavily and velocity slow to a crawl.

And the code will be good.

[rsalus]

not necessarily, TDD has little bearing on output quality

[bfeynman]

In what world or frame of reference would doing TDD have "little" bearing on output quality? If you build a system around satisfying some set of requirements it seems logical that output quality would have pretty heavy correlation.

[emigre]

It's possible to satisfy a set of requirements with code that's low quality. There's the maintainability of the code, for example, or the performance of the system.

[mpweiher]

The set of requirements TDD encourages code to meet happen to be ones that increase code quality.

Code that is easy to test tends to be well-structured.

Code that is badly structured tends to be hard to test.

TDD is not a QA methodology, it is a design methodology. It also tends to help quality out a lot, but that's a secondary effect.

[rsalus]

for LLMs, TDD amounts to little more than ceremony. there is a study on this exact topic: https://arxiv.org/pdf/2602.07900

[manmal]

That’s an interesting proposition, are you saying people do TDD just for the heck of it?

[jzig]

Pattern-based testing can theoretically reduce the token cost?

[reg_dunlop]

But that repurposing/removal is exactly what's avoided if you follow through with the SEF framework he outlines.

I have to push back on the idea that token costs balloon when using TDD within the context of a strong framework such as Jason has laid out here.

If the feature is repurposed/removed/refactored....I'd argue the specification wasn't well thought out prior to burning into tokens.

We're so eager to do a lot of the wrong things quickly, when it may serve us better to do a more precise thing slowly.

[zuzululu]

You cant spec out what you dont know, scope, requirements change from real world feedback

[reg_dunlop]

Then adjust the specs with the scope and feedback.

I fail to see the argument you're making...

Features aren't made in a vacuum. If specs are made/written....with the information available now...then it's better than not writing specs.

Writing specs with incomplete information is better than not writing specs with incomplete information.