[sethish]

I'm not sure who I would trust, but I don't generally trust university research departments to produce high quality, reusable software. Not that industry projects produce consistent quality software either, but fewer of those projects are open sourced.

[TimSchumann]

While I agree with your, let's call them reservations, about the quality of said software written by scientists who are not professional software engineers -- I still believe it would be a net benefit to the world if Congress tied all federally funded research dollars to an open source license, up to and including disclosure on hardware specifications and firmware source code.

I've got a friend on the data science/visibility team at CERN, and she has some fun stories. It's absurd to me that the reproducibility path for so much modern day research in nearly every field passes through proprietary closed source software, proprietary pre-compiled signed binaries, ditto for firmware, and then (sometimes) hardware where the entirety of the documentation consists of 'A service contract with the vendor'.

The reproducibility crisis in the sciences right now is essentially an excess of inputs and deficit of testing those inputs -- closed source hardware/software is not helping.

[peterwoerner]

Sure, but there is no reward for research departments to produce high quality code. Their job is to experiment and publish, maintainable bug free(unless it effects their code) doesn't get rewarded.

With that being said there are tons of high quality open source scientific computing projects e.g. lammps, abinit, dakota, Deal.II which are open source, ran by people at government research labs and universities and funded by e.g. NSF which work really nicely and are (I assume) quality software products. I will note that the Europeans seem to be better and producing these kinds of works than the Yankees.

[sitkack]

One wants to get something that is worthwhile, of course. It would be fairly easy to vet existing projects and codebases for their quality at a high level. Mostly at a binary level, with the goal to weed out total junk.

Like a license, I think it would be nice if projects adopted a development guideline, code formatting, the commit, code review, merge process, etc. So that everyone can have a common set of known operating norms around how that project is developed. It would be a lot easier for a funding organization to review if that were in place. The Robert's Rules of Code Review. Bonus points if this were all encapsulated into an automatic process.

I think there is a ton of value in funding projects (Apache,Rust,Blender,etc) as well as individuals. It would be wonderful if someone could go on sabbatical, full or part time and get paid to work on OSS. Maybe you apply for a grant, show them the git repos (process above analyzes), one has a plan (fix bugs, new features, evangelism, tPM, etc) for the time and with time with the bare minimum of goals. Like an agent can watch your activity log and do a roll-up.

I think 50k a year would be a number that many technical folks making much much more would jump at the chance to just work on OSS.

[acdha]

> Sure, but there is no reward for research departments to produce high quality code. Their job is to experiment and publish, maintainable bug free(unless it effects their code) doesn't get rewarded.

This isn't completely true. I'm familiar with at least few projects which got grants specifically to build tools for other scientists and those were great for funding general software engineering jobs not tied to the usual academic publication/job cycle. That was good for supporting ongoing upgrades — nobody gets a publication for upgrading a dependency, which is how you end up with someone keeping a Windows 95 box on life support for years because they were focused on science – and also for supporting things like good documentation since that's a serious time commitment.

It'd be an idea to scale up things like that for identified high-impact projects, with the trick being finding someone capable of overseeing software engineering rather than the usual academic focus of existing grant panels. Seems like something which might be a fit under the Commerce department since there's a lot of stuff which American businesses benefit from but generally do not directly support.

[peterwoerner]

You're a right, it isn't completely thankless. And it was suggested to me as a great way to increase citations was to write my algorithm to work in LAMMPs. Nevertheless, I looked at the risk-reward (and also considered the algorithm I was working on worthless and decided to punt).

[acdha]

Oh, I completely agree. The projects I saw were considered novel for actually getting funding for staff engineering positions which didn't follow the regular academic track. They had to do that to get the right people since the grad students were heavily incentivized to focus on things which would get noteworthy publications and you couldn't get experienced software engineers on the academic pay scale unless you hit the unicorn of someone who was really interested in the domain and didn't like money.

[ska]

There is plausibly an easy enough solution for this, if the grant structure allows/encourages departments hiring developers to help with this. If your primary source of labor for this is short term grad students with no expertise in software development, and not training mechanism, the results are predictable.

Another thing that works against this is a lack of academic credit for doing,overseeing, or steering the work. This also could change.

Changing the incentives can easily change the results.

[webmaven]

> Another thing that works against this is a lack of academic credit for doing,overseeing, or steering the work. This also could change.

> Changing the incentives can easily change the results.

Arguably, besides the PI (Principal Investigator) listed on a paper, you should also list a Principal Engineer.