Hacker News new | ask | show | jobs
by CuriouslyC 1454 days ago
Or run a partial suite...
1 comments

Partial suite is certainly an option. You can build tools that only test code that has changed and whose changes can have cascading effects on other code (i.e dependent code). Do you know if such tools exist, or would I (as in the software engineer working on the codebase) have to build them?
This is actually the default result you get if you leave the defaults alone in some CI systems! My current company runs CI this way, in fact. A checkout is re-used between builds, and Gradle computes the minimal set of tasks that need to be run based on changed files just as it would locally. TeamCity will set things up this way by default. If you want a clean build on every run, you have to opt-in by checking a box.

How well does it work? Well .... it sorta mostly works. Sometimes a build will fail for inexplicable reasons because Gradle/Kotlin incremental builds don't seem fully reliable. You re-run with a clean checkout and the issue goes away.

How much time does it save? Also hard to say. Most of the time goes into integration tests that by their nature are invalidated by more or less any change in the codebase. That's not exactly incorrect. Some changes in some modules do avoid hitting the integration tests, though.

TC can also do test sharding in the newest versions when you use JUnit. We don't use this yet though.

Partial is the most valuable when you have red tests that are resisting simple fixes. I would definitely encourage running the entire tests every hour or two at the least.

We have a project where the team split the tests into chunks and eventually I figured out the reason why is because they had coupling between tests, and running them all together ran into problems. I worry about other people opening that door, because it’s damned hard to close again.