[emptysea]

when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code.

At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket will be created for the test owner to fix their test (and it will be disabled for future test runs)

I think for small to medium projects, you can have all tests succeed but once the repo is large enough / has frequent enough changes, flakey tests are bound to slip in.

[Gigachad]

Our setup just reruns the tests a few times which sorts out flaky tests. The page then shows the most frequently failing tests so they can be properly fixed.

[wpietri]

That sounds like it works, but shouldn't flaky tests be detected by frequent automatic test suite runs?

[emptysea]

yeah I think the system is a combination, although I'm not sure the intricacies tbh

I've heard google does something fancier where they take a test, run it a bunch after it fails to check if it's failed

I think the system at work only runs each test a couple times before giving up and marking it failed

[jamesfinlayson]

I think GitHub does something similar - public website tests must always pass but if you break GitHub Enterprise you get three days to fix it (or something like that - I think they had a blog post on it).

[idontwantthis]

That sounds fantastic. What tool are you describing?

[emptysea]

It's some home grown UI glued on top of Jenkins

This is pretty much the one feature that's nice, otherwise it's like a worse version of circle/github actions