Hacker News new | ask | show | jobs
by 8organicbits 1607 days ago
> When an E2E test is failing consistently and nobody cares to fix it, that means the test isn’t useful. There’s no point in having it around.

I suspect this is a good idea, but it raises some red flags for me. People may not want to fix tests if they don't feel like they have time, or fixing tests will help their promotion (i.e. culture). Of course if you have good engineering culture, this is probably a useful signal for tests to remove.
2 comments
domesticsimian 1607 days ago
I think maybe the point of that quote is that failing tests are just noise. Either fix them or remove them. If I try to run all tests for some pull request and 20% fail, what does that tell me? In the case where we regularly have some number of continuously failing tests, it doesn't tell me much. Did my PR make it worse? Did my PR make it better? Having continuously failing tests definitely doesn't add value and explicitly makes things harder to reason about when looking at test results.
link
avensec 1607 days ago
One of the reasons we test is for confidence. If we can't trust a test, it isn't providing value. It may give negative value due to the time required to inspect the failure or general erosion of trust in the test suite.

One pattern that we can apply to increase visibility or ownership is stability metrics. If a test must/should be fixed many times can be teased out once you can view these metrics. On failure, display that this test has passed in this configuration for the past x-amount of runs. - Pass the last 100 runs? High likelihood the test is highlighting a bug and must be engaged on. - 95% pass rate in the last 100 runs? It may be time to quarantine this test and add it to the remediation backlog. Your level of acceptable false-positive rates may differ depending on team velocity and suite runtimes.

"How many tests are in quarantine, what is the average time-to-fix, and what direction is this trending" are valuable metrics that we can utilize to find ownership and highlight the technical debt.

As you said, culture around such patterns isn't always there.

link