| I generally don't write unit tests, and very rarely write system tests. Partly this is because of the type of stuff I deal with (Django applications), partly it's because I generally have access to good QA people, but mostly it's because the kinds of errors I see would not be caught by the kind of testing I can write in any reasonable amount of time. Here are some real world recent errors I've had to deal with, where testing wouldn't have saved me: - Poorly documented external API returns a redirect instead of a 404 in some cases. - Missing <title> tag in a random HTML template. - Poorly assembled SSL certificate chain for an HTTPS service fails for some browsers, but not all. - Celery tasks taking longer to process because there are now more things to do as more users sign up. - Starting a DB transaction in the wrong place causes more failures than necessary when only some of the modified rows cause constraint violations. - External API returns 0 sized file with no errors instead of the correct response, depending on time of day and phase of the moon. - CSS issue rooted in poorly set up position and z-index properties causes elements to be mis-aligned. - Missing clear: both; - Database server's disk filled up with log files from a rogue service. - Hosting provider gets DDoS'ed and their mitigation software starts returning random site redirects. - Namecheap's DNS gets DDoS'ed and the site is inaccessible. These are not the types of things that are easy to test, but very easy to verify by hand or catch at runtime. Personally, I am much more in favor of design-by-contract and logging and alerting of bad conditions at runtime. That's not to say I wouldn't test a financial trading algorithm, or a binary search library function. One of my projects (LogHog, a much easier to use syslog-type-thing for Python), has lots of tests because it's got a very simple and well-defined interface. It's almost library code and verifying its correctness is both easy and useful at the same time. But not all projects are like that, and sometimes easy testing is not actually useful and useful testing is cost-prohibitive. tl;dr: I can test the obvious stuff, but I can also just spend more time writing it more carefully. It's the non-obvious stuff that breaks: disks fill up, external API's misbehave, bad CSS, etc. You will never have "100% code coverage" because you are not testing all the stuff that actually affects your system, and your users don't care why the application is not working, they just care that it's not. |
"sometimes easy testing is not actually useful and useful testing is cost-prohibitive."
While I agree here, too, I think that the situation where testing is not useful happens very rarely in practice. Everything can break, and when you think it cannot be possibly be wrong now, a future regression might occur.
I think badly written tests are a different problem. I.e. tests that for example assert on the serialized string and hence take a lot of maintainability effort to adapt to changing production code. But then there is nothing wrong with the test per se, its just badly implemented.