Hacker News new | ask | show | jobs
by avensec 1611 days ago
One of the reasons we test is for confidence. If we can't trust a test, it isn't providing value. It may give negative value due to the time required to inspect the failure or general erosion of trust in the test suite.

One pattern that we can apply to increase visibility or ownership is stability metrics. If a test must/should be fixed many times can be teased out once you can view these metrics. On failure, display that this test has passed in this configuration for the past x-amount of runs. - Pass the last 100 runs? High likelihood the test is highlighting a bug and must be engaged on. - 95% pass rate in the last 100 runs? It may be time to quarantine this test and add it to the remediation backlog. Your level of acceptable false-positive rates may differ depending on team velocity and suite runtimes.

"How many tests are in quarantine, what is the average time-to-fix, and what direction is this trending" are valuable metrics that we can utilize to find ownership and highlight the technical debt.

As you said, culture around such patterns isn't always there.