[mariefred]

Flaky tests are indeed a big issue, the main concern being loss of confidence in the results.

The otherwise good advice for randomization has its drawbacks-

- it complicates issue reproduction, especially if the test flow itself is randomized and not just the data

- the same way it catches more issues, it might as well skip some

Something else that was mentioned but not stressed enough is the importance of clean environment as the basis for the test infrastructure.

A cleanup function is nice but using a virtual environment, Docker or a clean VM will save you a lot of debugging time finding environmental issues. The same goes for mocked or simplified elements if they contribute to the reproducibility of the system- a simpler in-memory database can help re creating a clean database for each test instead of reverting for example

[AstralStorm]

Sometimes it's the code that is flaky and not the test.

In case of concurrent execution there are a only a few reasonably working tricks like Relacy and other exhaustive ordering checkers as well as formal proofs. Neither is cheap to use, so you will always get flaky tests there - or rather tests that do not always fall.

[mariefred]

if the code is flaky then I have earned my pay honestly, this is a problem that should be solved.

Subtle concurrency issues are indeed very difficult to be found debugged and reproduced and randomization could help with that simply by covering more space.

[roland35]

I agree I think a large majority of flaky tests, for me at least, stems from some variability in the initial conditions of the test. It is good to uncover all the dependencies!

[AstralStorm]

If it's the test that is flaky. But it means that if production is not extremely consistent, you will see these effects live. Better handle them correctly.