[dmos62]

Unit testing usually isn't easily replaced by manual testing. If you have, for example, 3 units that can be in 2 different modes each, that's 2^3 different combinations, but only 2*3 unit modes. Testing the end result is more work than testing the units.

[dekhn]

Discovery science is different from web software engineering. Most discovery scientists use manual testing, not unit testing. Very few actually do integration tests or system tests (this is something I'm trying to change).

And, given the external results of the application, it's unclear to me how much additional value would come from a rigorous testing system.

[dmos62]

> Very few actually do integration tests or system tests (this is something I'm trying to change).

Care to expand on what you're trying to do?

[dekhn]

Sure, I'm trying to take the idea of merging continuous integration with workflow/pipelines. It's all stuff that I learned at Google and is non-proprietary. The idea is have presubmit checks that invoke a full instance of a complex pipeline, but on canned (synthetic or pseudoanonymized or somehow not directly connected to the prod system) data, as an integration test. This catches many errors that would be hard to debug later in a prod workfflow.

In a sense, I see software testing/big web data and modern large scale data processing in science as a continuum and I want to bring the practices from the big web data and testing fields to bear on science pipelines.

[dmos62]

Apart from a shift in mental attitude, is it primarily about getting a dataset for the integration test?

[dekhn]

also making sure the testing is hermetic (not breaking prod) and all the components are actually reproducible.