[forgottenpass]

I have a similar problem but unfortunately the conventional wisdom to long-duration tests is that we're just doing it wrong. I'd love to say there is a good system available for having a pile of infrastructure in a corner somewhere running tests, but I haven't found anything with out of the box support for a more complicated execution model than running latest tests on latest build artifact.

The best I've been able to accomplish is a set of parametrized Jenkins jobs that default to running latest tests on latest code, and report results to a custom external application. The test monitor has a status console and can automatically schedule more Jenkins jobs using build parameters to run the tests with specific versions of application and tests to bisect failures. It's ugly, but is tolerable for what we need.

[jonstjohn]

Agreed about the conventional wisdom. Unfortunately, I think many companies aren't 'doing it right' and end up with somewhat bloated systems. From what I'm hearing, most companies roll their own solution to that problem and don't use any specific tools to address it. I'm seen some rudimentary statistics in Sauce Labs, as well as built into Jenkins.