[amzans]

Your point of view resonates with me.

Over the years, I've learned to test "what a thing is supposed to do, not how it does it". Usually this would mean to write high level tests or at least to draft up what using the API might end up looking like (be it a REST API or just a library).

This approach comes with the benefit that you can focus on designing a nice to use and modular API without worrying on how to solve it from the start. And it tends to produce designs with less leaky abstractions.

Of course YMMV.

[finaliteration]

I just updated our “best practices” documentation to include the recommendation that tests be written against a public API/class methods and the various expected outcomes rather than testing each individual method.

I think the latter gives an inflated sense of coverage (“But we’re 95% covered!”) but makes the tests far more brittle. What if you update a method and the tests pass but now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed?

I like to think I’m taking a more holistic view but I could also be deluding myself. =)

[newfoundglory]

Why doesn't your best practices document include a requirement to run the whole test suite?

[finaliteration]

Ah, I guess that wasn’t clear. Typically, yes, all tests would be run, though as someone else mentioned sometimes during development a smaller subset may be run for the sake of time (but all tests would still be run prior to QA and final release).

[dllthomas]

In some contexts there may be valuable tests that take long enough to run they should be run out-of-band. That said, I don't see where the parent says they don't run the whole test suite.

[newfoundglory]

> now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed

[dllthomas]

Ah, yeah, not sure how I missed that. Maybe they meant the test for the broken method isn't run because it doesn't exist, but I very much agree that that's not the most natural interpretation.

[finaliteration]

Sorry, this wasn’t totally clear and I thought “run all tests” was implied. What I was trying to get at was the difference between “we have a suite that includes individual tests for every single separate method so we have great coverage” vs “we have a suite of tests that run against public APIs that still manage to touch the methods involved”. The former may test “everything” but not in the right way, if that makes sense.

I should have said “you’ve only written a test” rather than only having run a test.