[throwaway21321]

we had something like this. Environment was flaky and expectation was you run the test a few times till it passes.... as expected most developers who can issue put requests with curl or something would only need to "run" that test once

[jpollock]

That's not how CI works? CI is the backend verifying that the tests pass, not that the developer has run the tests.

From what I understand, requiring developers to assert they run the tests and not run them again from a clean checkout before pushing to prod is likely a compliance problem (SOX, PCI, etc).