[hbrn]

There's this stupid dogma: tests are good, codebase without tests is bad. It is incredibly hard to fight the idea of "we need more tests" because of that. Anytime engineering team gets some freedom, tests are usually the first thing they decide to work on.

But tests aren't inherently good. Good tests will speed you up, bad ones will slow you down. And it is incredibly hard to write good tests. Might be even harder than writing software that tests supposed to test:

If software works in 80% of cases, it creates 80% of value.

If tests work in 80% of cases, they cause harm in the remaining 20%.

[malermeister]

In my experience tests also slow refactoring down, despite the common mantra claiming the opposite.

Want to extract some logic into a new interface or introduce a new parameter to a method?

Well, now instead of just doing that, you will also have to update all tests dealing with said logic.

At my previous job I spent much more time updating tests testing trivial shit than actually writing useful code.

[hbrn]

Exactly my experience. Numerous times I found ways to reduce accidental complexity in software. But guess what, my change breaks 50 tests, because accidental complexity happened to be under test. What I anticipated to be an enjoyable 2-hour refactoring turns into 3-day chore. I guess I'll just leave the complexity in there.

Sure, those were "bad" tests. Accidental complexity is not supposed to be tested. And yet I had this experience in every single company I worked at. Maybe I'm just incredibly unlucky. Or maybe there's something wrong with "tests are good" mantra.

"Write tests. Not too many. Mostly integration"

[ludston]

Tests make refactoring slower, because you might have to also update the tests. On the other hand, in a complex enough code base, refactoring is impossible without sufficient test coverage. You can rely on the tests to catch edge cases that you weren't aware of in areas of products built on top of architecture you are maintaining.

Sometimes, all 50 of those stupidly over-complex tests that people have copy-pasted around are garbage that get in the way of your refactor, but sometimes one of them is protecting some important functional requirement that you didn't think about like "Test that this code called from a tight loop doesn't query the db hundreds of time which would lead to the startup time taking minutes rather than seconds".

[hbrn]

> Tests make refactoring slower, because you might have to also update the tests

No, tests can make refactoring slower, because they often test implementation detail instead of interface. Being able to distinguish the two is not as easy as it sounds, typically it requires deep understanding of the domain, which people writing tests might not have yet.

> in a complex enough code base, refactoring is impossible without sufficient test coverage

Sure, and in my experience complex code bases are typically complex due to the amount of accidental complexity, not because they actually solve complex problems. Like layers of overengineered classes intertwined through dependency injection for the sake of being more... testable. See the irony?

And indeed, refactoring becomes quite hard. You have two options to make it easier:

1. Identify and remove accidental complexity.

2. Cover it with tests and set the complexity in stone.

It's like fighting cancer with painkillers. It might look like it's working, but you going to die pretty soon.

> Test that this code called from a tight loop doesn't query the db hundreds of time which would lead to the startup time taking minutes rather than seconds

Oh, this is a great example. I saw the exact same thing implemented couple years ago. The whole team hated it because it caused tests to break constantly. And yet nobody on the team had the balls to remove it, and they were even following the pattern for new tests, because past tests had it.

If startup time matters for your app, there will be plenty of signals apart from tests that it's broken. If startup time doesn't matter, why are you testing it?

By the way, you haven't even noticed it, but you conflated startup time with number of queries. You're testing the wrong thing. What if hundreds of queries are still blazingly fast (due to being simple and/or cacheable) and have negligible effect on startup time? What if existing startup time is already extremely high (e.g. you're loading ML models in memory)?

If startup time truly matters for you, you shouldn't be testing it, you should be monitoring it. And I'm not even talking about some complex monitoring systems. When someone on your team says their developer experience got worse, that's the type of alert you need to listen to.

[mbrodersen]

Not my experience at all. I can only safely refactor major parts of the large complex C++ software I am maintaining because of the tests.

If you can’t refactor your code without breaking tests then your tests are testing implementation details and not higher level specification details. Your tests should test your use cases not your implementation details.

In other words, if you can’t refactor your code without breaking your tests then you are doing automatic testing wrong. It’s that simple.

[avl999]

Software isn't written once and left to run in production for it's lifetime, it is constantly being iterated on and changed. And a lack of tests will inevitably result in a slowdown of the rate you introduce new features as now you need to spend longer convincing yourself that the change you made didn't inadvertently break something else (plus rolling back and outages that will result from such an event).

Most tests written by people working on the product will of some baseline level of quality and even if they weren't, even if they were "bad tests". It is much easier to fix bad test than it is to deal with an application on a day to day basis which has no tests.

[hbrn]

> lack of tests will inevitably result in a slowdown

This is exactly the mindset I'm talking about. Lack of which tests? Manual? Unit? Functional? Integration? End-to-end? Doesn't matter, right? All tests have zero cost, the more the merrier.

And when exactly does the slowdown happen, on day one? Or maybe some time in the future? Do we have a way to tell when exactly should we write tests, how many, and which ones? Most of the time the mindset is "we don't have tests, therefore we need tests". Cargo cult at it's finest.

> Most tests written by people working on the product will of some baseline level of quality

Not in my experience. Tests are often introduced before you have a good understanding of the domain (hello TDD). Epitome of that is hiring junior engineers, realizing you don't want to assign any real work to them, and forcing them to write tests.

> It is much easier to fix bad test

Typically bad tests should not be fixed, they should be thrown away. But pretty much every company sucks at deleting bad tests, because there's the illusion of value they provide.