[amelius]

> there is a disproportionate focus on paper quality over "project quality"

One problem is bitrot. Stuff that runs now is not guaranteed to work in 1 or 2 years, let alone 10 years.

Even more so when it runs on fancy hardware, like GPUs.

[prepend]

This is one of the main reasons to require source release. Open source software is much more likely to run in 10 years. It’s actually useful to package everything together into a container or VM so all the packages are there too.

I work with some genome guys and they have this problem as their sequencers basically turn over in a year or two the advances are so fast. So they have to maintain the specimen as well as all the software versions they used for analysis. It’s a pain, but otherwise nothing is reproducible.

[sitkack]

The work should reproducible from not just artifacts, but also from a container. Sourcing compilers, libraries, etc is almost impossible. The NSF should really be running an archive and cluster for housing reproducible research that remains executable far into the future.