[TotempaaltJ]

I love learnings on automated testing. From the perspective of someone who isn't used to TDD or even just building many tests, maintaining E2E tests often seems extremely cumbersome. I wonder if I'm just missing out on the best practices, or if the tooling simply hasn't evolved enough yet.

[martinald]

The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem.

Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your own code. Adding tonnes of unit tests to that is sort of missing the big picture - you need to test it all works together as a user would.

[sidlls]

That's a rare attitude to have in the Bay Area anyway. Unfortunately. Everyone wants lots of unit tests because they run fast and give (roughly) instant feedback. Unit tests paint an incomplete picture. Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.

[mgkimsal]

> Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.

From my POV and experience, the middle ground is often what people refer to as 'integration' tests. Testing (without a browser), hitting endpoints/urls with known payloads and getting expected results catches errors with assumptions made about the interaction between various individual libraries.

At least in the web app world, my views are:

1. Testing the individual libraries gets you one layer of confidence. 2. Testing the interaction of those, usually via URL endpoints as various identities, gets you another layer of confidence. 3. Testing with E2E exposes primarily UI/JS problems.

When the first 2 are strong/solid, you can focus troubleshooting problems in #3 at the client/JS level first. It's not always the case, but it can help reduce concerns about "is this a back-end issue?".

I've been (slowly) trying to write more js component tests (in one case, with jest and vue), as it makes it easier/faster to test many permutations of input/validation/etc all at once. It's yet another 'confidence' area such that, when there are E2E tests, I can narrow down focus even more.

On a couple projects I've been on the past few years, we've found very few problems via E2E tests alone, mostly because there are so many back-end unit and integration tests. The E2E issues that are found are often UI-only (error state changes not rendering, sometimes perf issues, etc).

[bluGill]

The problem I have with unit tests is they inhibit refactoring off the API they test.

Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a while the company library implementers.

For everyone else we are writing to a business requirement that isn't well understood and may change. However the purpose of a unit test is to assert that no matter what this won't change. So every time you want to make a change all those tests are in the way of the change and need to be fixed.

Everyone writing tests needs to figure out their own middle ground. Because end to end tests have their own problems.

[Jtsummers]

That stretches the idea of a refactor, though. If a refactor changes the external behavior it's not really a refactor, which is a structural change. Unit tests when applied at an API level (which may be a very small unit or up to the level of a library, but that also stretches the definition of "unit test" depending on the size of the library) are there to ensure that changes to the internals don't impact the behavior.

As soon as you start changing the behavior, you have to change the unit tests. If you're adding behavior, you have to add tests. If you're removing behavior, you remove tests. If you're changing the way a procedure works, you change the related unit tests.

Really, any behavior change requires changes to the tests (whatever level they may be, if you want a high degree of test coverage).

[mgkimsal]

And without relatively comprehensive tests, you can't ever tell if a 'refactor' (in the structural sense of the word) actually worked. Did you change internal implementation code without affecting the consumer output? Without tests, you can't reliably tell with a high degree of confidence.

[bluGill]

Units by definition are not external behavior. sometimes they change external behavior, but there are a lot of changes that make code cleaner without changing external behavior. All too often I've discovered after a few years that I really need to split some unit into two.

[Jtsummers]

To clarify, the unit's own behavior is indeed not external to itself. But the way it behaves from the perspective of a user of the unit is external behavior. Changing the behavior, from that perspective, of a unit will necessarily invalidate its tests (some at least) and is not a refactor. That was my point, if you are changing an API you are not refactoring, you are changing its external behavior.

But this isn't just a problem with unit tests, it's a problem with all tests. And really, it's not a problem. It's the cost of doing business. Anything in flux (where that change changes its behavior) will force you to change anything that depends on it, whether it's a test or a user/client. If you aren't willing to discard the tests then the tests "freeze" the system in place, regardless of their level or kind (unit, integration, end to end).

[eatonphil]

Yep same. For libraries unit tests are great. For applications though I feel the most value writing integration tests and e2e tests. That's what helps capture the biggest user-facing bugs.

[Afton]

It's about tradeoffs. On one end you have precision, speed, reliability, diagnosability. At the other end you have "realness".

Unit tests fall on the far left, workload tests/E2E tests/testing-in-production fall on the far right.

It turns out that there's no 'wrong' level, there's just different tradeoffs. I've worked at a lot of companies that embraced the realness of E2E tests, but then suffered from the maintenance/performance/diagnosability/instability of those tests. I have colleagues who worked at places that avoided E2E at all costs, and suffered because they would have a green test run, but user scenarios that a simple E2E test would have caught, were completely broken.

IMO there is a lot that can be done to improve E2E testing at most companies, but they definitely have the capacity to add value to your release/testing pipeline.