[joeyo]

  > Iteration itself isn’t inherently bad. It’s just that the objective
  > function usually isn’t what we want from a scientific perspective.

I think this is exactly right and touches on a key difference between science and engineering.

Science: Is treatment A better than treatment B?

Engineering: I would like to make a better treatment B.

Iteration is harmful for the first goal yet essential for the second. I work in an applied science/engineering field where both perspectives exist. (and are necessary!) Which specific path is taken for any given experiment or analysis will depends on which goal one is trying to achieve. Conflict will sometimes arise when it's not clear which of these two objectives is the important one.

[jiggawatts]

There is no difference between comparing A versus B or B1 versus B2. The data collection process and and the mathematical methods are (typically) identical or subject to the same issues.

E.g.: profiling an existing application and tuning its performance is comparing two products, it just so happens that they’re different versions of the same series. If you compared it to a competing vendor’s product you should use the same mathematical analysis process.

[gwd]

I was kind of scratching my head at what GP was getting at as well; I suspect that "better" has a different metric in the second case: i.e., the scientist is asking which chemical A or B has the stronger desired medical effect; the engineer is assuming we're going with chemical B, and trying to drive down cost of producing the chemical or improve lifespan of the pills or decrease discomfort administering or increase absorption speed or tweak the absorption curve or something like that. Those metrics are often much easier to measure than the effectiveness of the chemical itself, and much less scientifically interesting.

[thyrsus]

This is how I perceived the difference: >SCIENCE< [a] create a hypothesis [b] collect all the data [c] check the hypothesis and publish; >ENGINEERING< [a] create a hypothesis [b] collect some data [c] refine the hypothesis [d] iterate over [b] and [c] until [e] PROFIT! (and maybe publish someday); the engineering approach is often better funded, allowing more data collection and better validation. If your engineering model is sufficiently deficient your product will be rejected in the market if it can even get to market. If your scientific model is sufficiently deficient, a researcher depending on that model will someday publish a refinement.