[trcollinson]

It's great that you have embraced testing in your product development! And rest assured it is not uncommon for you to run into issues when your test suite becomes large.

Unfortunately there is no silver bullet to make your tests run faster. I have used both Jenkins and CircleCI and they are great for most environments.

To combat this speed issue I have added a timer to all of my individual test runs. In Ruby with rspec, for example, this is relatively simple by just adding --profile to your .rspec file. Every time your test suite is run, this will show the top 10 slowest running tests. Each testing framework has a similar mechanism or add-on which will give you a report. Next start refactoring those long running tests. At first it will seem like a slow processes, but working on it a bit each day will quickly show significant improvement in your testing suite speed.

Next, watch out for tests, particularly integration tests and Selenium tests, which either test the framework and not your code, or which are redundant. A number of times I have walked into clients offices and have been told that their test suite is as "tight as it can be" and yet still runs slow. I look through it and find many areas where engineers in their desire to test thoroughly were unfortunately testing areas they don't need to. A bit of refactoring again showed massive dividends.

Finally, don't be afraid to segment your tests to run only portions that are effected by the code that was changed. It's true that running the entire suite is important, but you're correct in only running the whole thing at certain times. My current project has segments for API, various Front End resources, and service integrations. There are a number of sub segments as well. Again I think of this as a test refactoring exercise, but it will pay off if designed well.

So, unfortunately I don't have a software solution that will solve your problems. But I can say that a few refactoring activities can make all of the difference.

[jonstjohn]

Great tips - thanks! In addition to some slow-running tests, we do have a lot of test duplication. It's probably time to step back and evaluate what tests are overlapping functionality.

Do you use any tools to facilitate your workflow on fixing tests? Typically, we run a regression build on a branch and may find a half dozen failures. With 4-5 developers on a distributed team, we've built a utility for 'checking out' and marking a test as fixed (to be verified on the next build) but haven't seen any similar functionality in the CI systems we use.

[trcollinson]

Nothing slows down a test suite like test duplication! I would definitely suggest a refactor, it will help. But remember you don't eat an elephant all at once, take it one bite at a time. A small amount but consistent level of test refactoring will go a long well.

It sounds like you have a good workflow to me but obviously you feel you have a problem or you wouldn't be posting here. I've working in environments where the failing tests are assigned out on an individual basis, where there was a bouncer who took on all broken tests, and where each individual engineer would run the tests often enough that they would just fix what they broke. The latter method is my favorite. But it is not always feasible.

I guess the answer really depends on your environment. I would be very happy to give my advise (for what it's worth). If you would like you may email me. My address is my Hacker News username at googles very large and famous free email service.