|
|
|
|
|
|
by dudzik
2198 days ago
|
|
|
Ruby is not really slowing us down, the bigger issue is that we have a lot of integration tests and few isolated unit tests. So we are massively parallelizing our builds so we can get p50 build times of 23 minutes when running the whole test suite. The time mentioned in the article is a little bit closer to the p95 time. > If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. That is a good approach and we also used to disable tests that would fail on full master builds. When it comes to fixing intermittently failing tests we are now storing every test run so we can analyze our test suite. This way we are able to find the slowest and flakiest tests in the codebase, disable them, and notify the owners of the code to investigate. |
|
|
In your experience, do you find high value in having your tests be 80/20 (rough numbers) integration/unit vs the other way around?
I guess it depends on the programming style, but here it’s the other way around. Some minority of engineers here clamor for end to end tests every once in a while, but we’ve rarely found the difficulty of establishing the test harness to be worth the marginal gains in reliability. We’re also wary of the temptation to write e2e tests to test unit-test-level correctness, because of the implications on test performance, like you describe.
On our side, the closest thing we have to ubiquitous e2e testing is that we spin up and spin down one ephemeral database instance per test module (could be used by many test functions).