[alkonaut]

How is the hard limit enforced? By the CI tool as a failed build?

It would be perfect if one could somehow keep the build quick using tools. A hard limit on a build might not be the best way. If the test is taking Limit minus one millisecond, and I add one new test that takes one millisecond, I break the build. But the culprit was really commit yesterday that added a test that takes 9 minutes. The worst thing one can do in CI is somehow flag the build as "not good enough" and point the blame the wrong way.

A tool that treated test quality/perf like any other asset and fails builds based on time limits etc would be perfect. It's really hard to do that especially on shared CI servers because of fluctuating performance.

[markoa]

Yeah sorry for not being clear there, by hard limit I didn't mean that CI should fail the build. But that the number should be a fixed threshold for the team to optimize for. Thanks for your feedback.