[chelmertz]

This is a really interesting problem area. Tests that fails sometimes are really annoying because of the "broken windows" analogy. Are you using the most unstable tests as input of what to redesign next? Is it kind of a "deal with it" situation, where you need to retry the test suite a couple of times per commit, until it becomes green?

[db48x]

The test suite doesn't get retried, we just wait for another commit to come in; it's not generally a long wait. See https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

Another problem is that running the tests takes ages.

[gcp]

You both "deal with it" and there's task forces that look at the tests that fail most often, and try to find someone with the skills to investigate.