[justinpombrio]

Yes, definitely that. I've also seen entire piles of tests be flaky because:

- They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree.

The general trend I've seen is that the more "e2e" a test is, the flakier it is.

[singingsolitue]

End to end tests definitely end up being flaky, especially in large systems. One level of testing would be unit tests, but e2e tests have their own place, where they do end to end sanity checks. In my experience at Rippling, we have managed to identify a lot of such flakiness by pure first principles reasoning of the behavior, and in most cases, it turned out to be a subtle bug in the code. As the org grows larger, there should be a team that just attacks flaky tests, either from a fix point of view by reviewing tests, or from tools point of view, where finding the gap becomes easier for the product teams!

[deckard1]

e2e is the number one productivity killer at my org, by a long shot. If there is such a thing as a non-flaky e2e test, I have yet to see it. That, or the test does nothing.

Selenium seems fine for the most part. It's a solid tool. Where it falls apart is when developers do not account for all the various ways the browser session will go wrong. There are simply too many variables at play. A/B tests, cookies, popups, network conditions, machine speed (and current load). Writing e2e is like being blindfolded and tying one hand behind your back while your coworkers take turns spinning you in your chair.

It constantly amazes me that companies think they can put more tasks on a developer's plate with zero impact to productivity. They really do believe that all testing is free and will somehow pay for itself. And yet a single QA human going through a simple testing plan will catch more actual bugs than hundreds of e2e tests that cost a literal fortune to maintain.

[pydry]

IME the source of flakiness in tests is always something that you can deal with given enough time and about 15-20% of the time it is a bug in the code itself, sometimes quite a dangerous one.