System-level tests are important, but they're not much help with refactoring code. For some applications, particularly in my domain of analytics, it's very important that we can do that effectively.
When you're refactoring code to reduce coupling, you absolutely HAVE to do it with system level tests.
Doing it with unit tests simply means that you'll end up writing the test, refactoring and then completely REwriting the tests AGAIN to get them all to pass because you're changing the method contracts and the objects being mocked.
Tests that fail every time you refactor are totally meaningless and a waste of time. They don't detect bugs. They just detect changed code.
> Tests that fail every time you refactor are totally meaningless and a waste of time. They don't detect bugs. They just detect changed code.
I agree. This is where I think the distinction between classical and mockist testing [1] is useful. These days, most TDD involves mocking or stubbing every single dependency, effectively turning your units into a white box - "isolated TDD." When one has code whose implementation is known by and manipulated by client code, refactoring will almost certainly break stuff.
Why would you want to liberally refactor code when you know you will break 10 of your tests and have to rewrite them?
One frequently sees hardcore TDD advocates patting themselves on the back for isolating everything, because... now they can swap the database for a third-party API, in-memory store, remote service, or whatever. Really? You're going to replace the database with something that has wildly different reliability constraints? And why would you ever need to replace your database with a remote third-party service? I'm sure it can be useful, but for most people, YAGNI. Perhaps I've merely not worked on enough Web Scale™ or Big Data™ projects.
>> When you're refactoring code to reduce coupling, you absolutely HAVE to do it with system level tests.
I think this comes about because tests are written against every class in your system. I find unit tests are far more useful if you focus on testing abstractions rather than every single class e.g. you have a reporting abstraction in your code, instead of testing every class used within that abstraction you only test the public API that you want to expose. This allows you to do black box testing which is infinitely better when it comes to refactoring, you should be able to restructure the internals of that particular API without having to change your unit tests at all.
My feeling is that a lot of the frustration with TDD at the moment is that people are writing tests for every public method in their system. If you focus more on the behaviour of your abstractions you gain a lot more freedom when refactoring and can greatly reduce the number of tests you write without reducing coverage.
>I find unit tests are far more useful if you focus on testing abstractions rather than every single class e.g. you have a reporting abstraction in your code, instead of testing every class used within that abstraction you only test the public API that you want to expose.
Yes, this is exactly what they're useful for. Unfortunately, if you have a big ball of tightly coupled muddy code and you're working on prying it apart and creating useful abstractions you can't use unit tests to get there.
The only way you can do test driven refactoring in that case is to create system level functional tests and then rework the code underneath them. Once you've got decent abstractions and a solid set of APIs and only then you can start writing unit tests against them.
The definition of a refactoring (at least the one I give juniors) is the modification of a system of code such that an external contract remains valid. An external contract is typically validated by tests (though it doesn't have to be). If your refactoring crosses a unit boundary, you're certainly going to have to test at a higher level (assuming you want the safety of a contract validation through tests). Otherwise you're modifying the contract of the system under test and you've crossed over from refactoring to redesign and reimplementation.
Doing it with unit tests simply means that you'll end up writing the test, refactoring and then completely REwriting the tests AGAIN to get them all to pass because you're changing the method contracts and the objects being mocked.
Tests that fail every time you refactor are totally meaningless and a waste of time. They don't detect bugs. They just detect changed code.