[yakult]

I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without my source code (preferably the whole environment) unless they spend a lot of time on trial and error like I did.

The tweaks I did to glue things together has no theoretical value and don't belong in the paper. As a practical matter, I can't fit a lot of source code into short paper format.

What do?

[erichocean]

Open source the code and supporting data.

[yakult]

It's not that simple. What if some of it is proprietary? What if I'm not allowed to submit code because I need to be anonymous so reviewers can maintain impartiality? What happens when one of the upstreams update and breaks my code? Do I need to keep it updated? Forever?

[adrianN]

At my institute at least, scientists are required to maintain everything that is necessary to reproduce a result for at least ten years. That includes all the data and the software used to produce the results. It's not an easy job, but it's important.

[yakult]

If your institute also mandates they make the data/software publicly available, then that's definitely the exception rather than rule. Also must be hideously expensive.

It almost never happens that a paper I read actually comes with usable source code.

[pixl97]

Then your results are not reproducible and your conclusion is suspect.

[chriswarbo]

A lot of thought has gone into such questions. For example, see the guidelines at https://www.epsrc.ac.uk/about/standards/researchdata/

[Klockan]

> Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without my source code (preferably the whole environment) unless they spend a lot of time on trial and error like I did.

Then leave out the results since they are just an anecdote. If you want to include experimental results then it has to be done in a scientific fashion.