[cratermoon]

At one place I consulted, the fte lead ignored flaky tests and attributed failures to the tests being wrong.

A few months later...

The code that was failing intermittently was found to be using floating point types for money. Yeah, I'm gonna wanna fix that.

[aidenn0]

Right if you have flaky tests there are 3 acceptable responses:

1. Fix the test

2. Fix the code that is being tested

3. Say "well we don't need this software to be reliable anyways so let just stop running tests"

But many places seem to adopt hidden option #4 "Run the tests and ignore failures"

A related issue is dialing the tunables for warnings up to 11 and then not reading any of the warnings. Once I saw a case where the build generated 1000s of warnings. Found a bug and said "this would be flagged as a warning even with relatively low warning settings" sure enough it was.

Obviously fixing warnings is good, but if they had just lowered the warning setting to be something reasonable, they would have had maybe 10 warnings total, one of which was a bug, which is a lot more useful than 1000s of warnings, at least one of which was a bug.

[cratermoon]

Option #4 is just option #3 but keeping the costs of running tests you ignore.

You're right about excessive warnings, but then sometimes note. Running `gcc -Wall` used to be considered madness, and if you did it now on a codebase that has been around a while and not kept clean, you'd drown in messages. The key is to turn it on from the very start and fix things when there are 10 warnings instead of 1000.

This decay happens with test suites, too. One or two tests start to fail, and instead of fixing them, people ignore the failures. A bit later, it's five tests, then 10, and pretty soon the programmers see the tests as broken instead of looking at the failures that let things get to the point where there are so many failing tests.

[aidenn0]

The fix for both situations is similar though; dial down the {warning strictness|number of tests run} until you get a clean {warnings|test-run} then enable them one by one in order of how easy they are to fix.