Hacker News new | ask | show | jobs
by ddevault 2008 days ago
Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.
1 comments

But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.
They say that contempt is the beginning of the end of the Rule of Law, which is why you should be careful not to pass frivolous laws.

Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times.

The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.

We actually have different classes of tests to allow for some more flaky tests. You definitely don't want to run those flaky tests after every build, but you should be able to eventually get a run where those tests pass before handing it off to customers.
Interesting. Doesn’t this imply that the behavior itself is flaky?
Right - but you fix the bug. You don't add a tolerance.