[finaliteration]

I just updated our “best practices” documentation to include the recommendation that tests be written against a public API/class methods and the various expected outcomes rather than testing each individual method.

I think the latter gives an inflated sense of coverage (“But we’re 95% covered!”) but makes the tests far more brittle. What if you update a method and the tests pass but now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed?

I like to think I’m taking a more holistic view but I could also be deluding myself. =)

[newfoundglory]

Why doesn't your best practices document include a requirement to run the whole test suite?

[finaliteration]

Ah, I guess that wasn’t clear. Typically, yes, all tests would be run, though as someone else mentioned sometimes during development a smaller subset may be run for the sake of time (but all tests would still be run prior to QA and final release).

[dllthomas]

In some contexts there may be valuable tests that take long enough to run they should be run out-of-band. That said, I don't see where the parent says they don't run the whole test suite.

[newfoundglory]

> now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed

[dllthomas]

Ah, yeah, not sure how I missed that. Maybe they meant the test for the broken method isn't run because it doesn't exist, but I very much agree that that's not the most natural interpretation.

[finaliteration]

Sorry, this wasn’t totally clear and I thought “run all tests” was implied. What I was trying to get at was the difference between “we have a suite that includes individual tests for every single separate method so we have great coverage” vs “we have a suite of tests that run against public APIs that still manage to touch the methods involved”. The former may test “everything” but not in the right way, if that makes sense.

I should have said “you’ve only written a test” rather than only having run a test.