[Miltnoid]

My research area has actually started doing that. Sure it's a CS discipline, but yeah we have a second component of our conferences where we automate our benchmarks, and other people run those benchmarks and validate the results.

[MaxBarraclough]

For CS publications, it should be the norm to insist on inclusion of all source-code and data-sets used, unless there's a compelling reason they can't be included.

It's absurd that reproducing results be any more of a challenge than simply re-running a freely-available program.

[TropicalAudio]

All but one program I wrote for scientific publications have been on the payroll of corporate grants, and the IP rights hooked right back into the company paying for the whole show. Can't exactly open-source things you don't actually own.

[MaxBarraclough]

Then you should be denied the opportunity to publish. Corporate obstinacy should not trump good science.

Software is unique in that it can be generally be duplicated and executed trivially.

There's no way to make it trivial to reproduce a test on the strength properties of a new ceramic. There is a way to do this for software, and it's rather silly that it isn't standard scientific practice to do so.

I realise I'm taking a strong line here, but I've never seen a good argument against it.

[TropicalAudio]

The unfortunate answer is that instead of harder-to-reproduce science, no science would have been published in all of those cases. Quite often, sharing knowledge is a secondary goal to achieving the set goals of a project. In niche fields, input from any source is welcomed gladly, as long as the papers are sound. Banning corporate players from sharing knowledge in your journal if that's the majority source for input in the field is not really an option.