[Bukhmanizer]

As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible.

If the problem was only unpredictability, then projects with a clear and defined end goal (eg, a website to host results) would be of substantially higher quality. But they’re not. Well defined projects tend to end up basically just as crappy as exploratory projects.

The problem is evaluation and incentives. There’s literally no evaluation of software or software development capability in the industry. I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money. Usually there are grant updating mechanisms, and reports, but he bsed his way through that knowing there’s a 0.0000000% chance that any granting agency is going to look through his code. The fraud was only found because he got fired for unrelated activities.

I once looked up older web projects on a grant. 4/6 were completely offline less than 2 years after their grants completed. For 2 of those 4, it’s unclear whether the site ever completed in the first place.

[matthewdgreen]

>I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money.

I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about. Intentional academic fraud is a very small percentage of what happens in academia. Partly this is because it's so stupid: academia pays poorly compared to industry, requires years to establish a reputation, and the systems make it hard to extract funds in a way that would be beneficial to the fraudster (hell, I can barely get reimbursed for buying pizza for my students.) So you're going to do a huge amount of work qualifying to receive a grant, write a proposal, and your reward is a relatively mediocre salary for a little while before you shred your reputation. Also, where is your "collected money" going? If you hire a team, then you're paying them to do nothing and collude with you, and your own ability to extract personal wealth is limited.

A much more common situation is that a researcher burns out or just fails to deliver much. That's always a risk in the academic funding world, and it's why grant agencies rarely give out 5-10 year grants (even though sometimes they should) and why the bar for getting a grant is so high. The idea is to let researchers do actual work, rather than having teams manage them and argue about their productivity.

(Also long-term unfunded project maintenance is a big, big problem. It's basically a labor of love slash charitable contribution at that point.)

[Bukhmanizer]

> I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about

This isn’t a rare edge case, this is very common in software projects. I’ve heard of it because I was part of the team brought in to fix the situation.

Intentional fraud only is rare when it’s recognized as fraud. P-hacking was incredibly widespread (and to some extent still is) because it wasn’t recognized as a form of fraud. Do you really think not delivering on a software project has any consequences? Who is going to go in and say what’s fraud, what’s incompetence, and what’s bad luck?

The problem is that the bar for getting software grants isn’t high, it’s nonsensical. As far as I can tell, ability to produce or manage software development isn’t factored in at all. As with everything else, it’s judged on papers, and the grant application. In some cases, having working software models and preexisting users end up being detrimental to the process, since it shows less of a “need” for the money. You get “stars” in their field, who end up with massive grants and no idea of how to implement their proposals. Conversely, plenty of scientists who slave away on their own time on personal projects that hundreds of other scientists depend on get no funding whatsoever.

[lazyjeff]

Just curious, what kind of 3-year informatics grant not being completed ends up with a team brought in to fix the situation? Multi-million dollar grants don't sound big enough to be a dependency for any major customer (like defense or pharma), so I imagine if fraud was detected, they would just demand a reimbursement and ban the PI.

But I think you're both right in some sense. The cases of intentional major fraud is probably a rare edge case and they make the news when they're uncovered. But there's a lot of grey-ish area like p-hacking as you mentioned, plus funding agencies know there needs to be some flexibility in the proposed timeline due to realities. Realities like you don't necessary get the perfect student for the project right when the grant starts, as the graduate student cycle is annual, plus the research changes over time and it isn't ideal to have students work on an exact plan as if they are an employee.

But I totally agree that maintaining software that people are using should be funded and rewarded by the academic communities. A possible way to do this is have a supplement so that after a grant is over, people who have software generated from the grant that is used by at least 10 external parties without COI, should be funded 100K/yr for however many years they are willing to maintain and improve it. Definitions of what this means needs to be carefully constructed, of course.

[Bukhmanizer]

I'll be a bit vague to protect my coworker's privacy, but the scientist was fired for other, unrelated violations, and my boss was brought in to replace him. I think he was leading an arm of a "U" grant, so he wasn't the only senior PI on it. Since they handled it internally, they couldn't just demand a reimbursement. On some level administration knew that the project wasn't moving forward, but once we started asking around, it was clear that there was no effort to start the project at all.

>But I totally agree that maintaining software that people are using should be funded and rewarded by the academic communities. A possible way to do this is have a supplement so that after a grant is over, people who have software generated from the grant that is used by at least 10 external parties without COI, should be funded 100K/yr for however many years they are willing to maintain and improve it. Definitions of what this means needs to be carefully constructed, of course.

I think that this is a great idea.

[qmmmur]

I can tell you why the sites went offline, because the funding stopped. I don't know what you're research background is but its painful to even get 5 GBP a month to host a droplet on digital ocean in a pretty lucrative department with liberal internal funding.

[Bukhmanizer]

Agreed, but all these little things are just a sign that the industry just does not give a shit about software. They could develop mechanisms to fund this stuff, pretty easily actually. But they don’t.

A couple of other weird inequities that I’ve found are: 1. It’s hard to get permission to spend money on software subscription based licenses since you won’t “have anything” at the end. However, it’s much easier to get funding for hardware with time based locks (e.g after 3 years the system will lock up and you have to pay them to unlock). The end result is the same, you can’t use the hardware after the time period is up, but for some reason the admin feels much more comfortable about it.

2. It’s hard get funding to hire someone to set up a service to transfer large amounts of data from different places. It’s much easier to hire someone to drive out to a bunch of places with a stack of hard drives and manually load the data on them, and drive back. Even if it’s 2x more expensive and would take longer. Why? Again my speculation is that the higher ups are just more comfortable with the latter strategy. They can picture the work being done in their head, so they know what they’re paying for.

[selimthegrim]

Louisiana state government spent a buttload of money on dedicated high speed fiber optic lines between a bunch of different universities in the state for videoconferencing, telenetworking, "grid computing" etc. 10 years later the only people who remember how to use the system are at LSU, rendering the purpose moot. Everyone else just uses Zoom or Skype.

https://www.regents.la.gov/assets/docs/Finance_and_Facilitie...

[mschuster91]

> The end result is the same, you can’t use the hardware after the time period is up, but for some reason the admin feels much more comfortable about it.

Simple: predictability. With a subscription based model, admin has to deal with recurring (monthly / yearly) payments, and the possibility is always there that whatever SaaS you choose it gets bought up and discontinued. Something you own and host yourself, even if it gets useless after three years, does not incur any administrative overhead and there is no risk of the provider vanishing. Also, there are no "surprise auto renewals" or random price hikes.

> 2. It’s hard get funding to hire someone to set up a service to transfer large amounts of data from different places.

Never underestimate the bandwidth of a 40 ton truck filled with SD cards. Joke aside: especially off-campus buildings have ... less than optimal Internet / fibre connections and those that do exist are often enough at enough load to make it unwise to shuffle large amounts of data through them without disrupting ongoing operations.

[pbourke]

Is N years of opex not part of the budget in grant applications?

[qmmmur]

In research no and it would depend entirely on your institution. For example, I looked at a job putting together a portal for people to freely examine the research put together for a research team. The project had secured a connection with the british museum, and so that website would live on under that. However, if the project had asked to host it themselves even for 60$ a year for 10 years the answer would be no. Funding grants see small opex that extend beyond the life of the project to be open to corruption or just too facile to fund, wrongly or rightly.

[unishark]

> As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible.

I've done both, and OOP can also make things worse. Now instead of just doing the calculations in a straightforward procedural fashion anyone who knows the research can understand, you've added a layer of structure to obfuscate it, and that structure may be harder to change if you guessed wrongly about what will be consistent and what won't. Research by its nature needs to be more flexible and will be more unpredictable than industry development. It is far more common to have to go back and reexamine even your most basic assumptions.

Of course a lot of researchers are doing the same things as industry (what should be described as development and not be getting research funding), and are certainly doing a much more amateur job of it.

Grant fraud is penalized severely in the US by the way. You can even get a bounty for reporting someone.

[marmaduke]

I work on a 12+ year academic (full stack Python) codebase, where there was an initial push for an OOP/DI architecture which was key to adapting to later grant requirements. The codebase is still evolving fine.

[paulclinger]

> I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money.

I wonder if a whistleblower payout similar to the one that SEC is doing for 1M+ fines (10-30%) would help in cases like this. The host organization would potentially be on the hook as well, so there is going to be a significant incentive to not let that happen (especially with all the associated reputational damage).