[ApostleMatthew]

My PhD adviser was a good friend of Chandler's, and a lot of my doctoral work was based on previous work done in his lab. I'm honestly really surprised one of his students/post-docs made a mistake of this magnitude -- generating a velocity distribution which does not follow the Boltzmann distribution is not something I'd expect to be a result of code coming from his lab. But we all make mistakes, and this really just lends more weight to the idea that all code used in producing published results should be posted along with the article itself, just like data sets often are (at least from studies with government funding).

[ur-whale]

Refusing to share their code falls under the "we all make mistakes" umbrella you think?

[whateveryou381]

Sharing code is not always the answer. Independent thinking implies independence, in code and more abstractly.

The real 'mistake' is allowing the drama to escalate to the point that it is toxic. People who are truly interested don't care about who is right and who is wrong.

When disagreements like this come up, having common and good test cases are probably the most important (and is indeed the way in which the problem, generation of non-Boltzman behavior, was found).

In the end ST2 itself is flawed, and Princeton admitted that the discussion of water was not significantly advanced through this drama. Is it worth it?

Understanding the argument and its importance should be the focus.

[YorkshireSeason]

   Sharing code is not 
   always the answer

Why let the perfect be the enemy of the good?

Sharing code is not the full answer, especially in case of Monte-Carlo simulations in physics, because that kind of algorithm is hard to test: What is the testing oracle? What is the specification?

But sharing code is part of the answer.

Setting up a culture where it is unacceptable to submit a paper without open-sourcing the code and suitable testing (for simple edge ases), and suitable scripts that make reproducing the software simulations easy, is good scientific 'hygiene'. See for example [1, 2] for efforts towards reproducible software submissions in computer science.

Reproduciblility is the very essence of the scientific method.

[1] http://evaluate.inf.usi.ch/artifacts

[2] http://www.artifact-eval.org/

[mannykannot]

In this case, independent thinking did come up with independent results, but the refusal to share code significantly delayed the recognition of which line of inquiry had been more successful.

You might want to be careful about using that "was it worth it?" argument - someone might reasonably ask, was it worth funding either team? If science doesn't show that it does good work, then its funding can be more easily questioned.

[s-shellfish]

Sharing code can be just as confusing as it is enlightening.

[mannykannot]

Ordinary text can be confusing. That is not a reason for not publishing, and one of the purposes of a journal's editors is to keep the confused and confusing stuff at bay.

[s-shellfish]

So, black box, closed source - e.g. history repeats itself.

[ApostleMatthew]

Oh, I meant making the error in the code fell under the we all make mistakes. Not the refusing to share it.