[flankstaek]

This letter feels as though it is overlooking a large point of contention.

>The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers

Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

[klenwell]

The conclusions from the first critical code review cited:

All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.

https://lockdownsceptics.org/code-review-of-fergusons-model/

[pjc50]

That's the most biased opinion imaginable by an astroturf group. There's no information on who they are, but the article currently on their front page is by Toby Young, who is a Spectator/Quillete guy and to be found backing almost all stupid ideas within British politics. https://www.spectator.co.uk/article/This-lockdown-may-kill-m...

[rscho]

Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health?

What could go wrong?

[SpicyLemonZest]

If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.

[rscho]

That's also true. Now the million $ question:

Whom would you like our society to rely on to generate quality work?

I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions.

There is no smart solution to a stupid problem. But the truth is that _as an institution_ the NHS is the only actor whose mission is to optimize towards public health.

[SpicyLemonZest]

In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good results is profitable!

[mturmon]

> In most fields, our society relies on private industry to generate quality work ...

This is just not true. Military, police, courts? What does "most fields" even mean?

[klyrs]

> it seems only fair to say that academics optimize for prestige and publication in good journals.

On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time.

On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles on the basis of their veracity (or, if incorrect, as punching bags). Of course that's a fallible process and just as vulnerable to cronyism, but when researchers are caught cooking the books they're discredited, not rewarded.

[friday99]

Authors aren't actually incentivized to find accurate results, but rather publishable results, which typically means novel. Researchers also cite articles based on their impact, not their veracity. There are plenty of instances of retracted results continuing to be cited as if they are still accurate.

There are issues with both industrial and academic research, but I do think that industrial research is more transparent in its motivations.

[toolz]

Everyone working for a living optimizes their work towards making more money. That doesn't change for people funded with public money. It's pretty widely accepted that the people in charge of public funding (politicians) sometimes act outside of the publics interest for self-gain.

I make no claim as to which achieves better results for the public because it's such a complicated problem, but I think it's rather naive to just assume publicly funded incentives are more aligned with social health than private incentives.

[spamizbad]

That last part is mindblowing given the fact that the health insurance industry in the US is trying to argue it shouldn’t have to pay for COVID-19 treatments because it’s part of a pandemic and not part of normal medical treatments.

I guess if your model is:

   if pandemic and COVID:
       is_covered = False

You can have a very clean pandemic model.

[lern_too_spel]

That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones.

The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.

[Joeboy]

> That was a shitty code review

Damn straight. The only sense in which it was not a shitty code review, is that it didn't actually review any code. Looking into the linked tickets is a time consuming faff, but people should actually do it before taking the blog post at face value.

[klenwell]

I agree. Moreover, the cited bug ("predictions varied by around 80,000 deaths after 80 days") doesn't really seem to impact the over all policy implications: no lockdown means exceptional number (400 to 480k) of deaths.

Unless the entire simulation is bogus, it comes off as nitpicking.

[thu2111]

Seriously?

Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct.

Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?

[lern_too_spel]

What does reading uninitialized memory or reusing variables that weren't set to zero have to do with seeding issues? Read my comment again and reply to its content instead of making up a comment that you would like to reply to.

[Nursie]

Yeah that's a weird set of conclusions there. I would think the papers based on it want another look though.

[lbeltrame]

Back in 2008-2009 my then PI tried this: it worked for a while, but then it proved to be untenable. Not because there wasn't enough money (there was) but because the university did not like the idea to get someone on board just to help researchers develop software, and thus there were so many roadblocks at one point that it was impossible to go on.

[btrettel]

I'm interested in hearing more details on this, if you're able to discuss them.

[lbeltrame]

I don't honestly know how much I can disclose (FTR, I was not the one doing that job - I was working as a postdoc there at the time), but I'll try to summarize it briefly:

- PI got a couple of EU funded grants

- PI wanted to build a program / programs out of some ideas he had before moving to the institution he was currently employed at

- PI hired a software developer (actually a software engineer) to do this job

- Statistician drafted algorithms, developer created the software

- I used the software for my Ph.D. thesis, got hired at the lab of the PI as a postdoc, new requirements arose due to the way I used it

- Developer made a preliminary version of a new version of the software following discussions with me

- University made it very hard to keep developer on the team, due to bureaucracy and kind of hostility towards this kind of employment

- At some point (I can't recall the details exactly, but it was something that spanned almost one year), the form of keeping the developer on board was no longer possible

- PI offers an alternative contract, but it is financially wasteful to the developer (not the fault of the PI, but the way certain things work in my country)

- Developer leaves the project

Also the university, to my knowledge, complained that the developer cost a lot (IIRC, the project was paid at market rate, so in line with other, non academic software projects).

I can't comment on the quality of the software we used (sadly it was never open sourced) as it was in Java and I only have a passing understanding of the language, but the approach of having a dedicated developer IMO worked (and also net quite a number of publications over that period).

[rch]

For this to happen there would need to be a nationally accepted PE certification for 'software engineers' on par with other engineering disciplines. It's unreasonable to expect non-experts, and experts from other fields, to perform credentialing on a case by case basis.

[vonmoltke]

There is no "nationally accepted" PE license for any discipline. It's a state-level credential and reciprocity is not universal or automatic.

[rch]

I included the "nationally accepted" bit since I'd prefer to see some incremental improvement over current models, but I don't have confidence that an international standard would be accepted in all relevant contexts. It could still be administered at the state level.

I'm also ignoring the requirement that candidates must start with a degree from an ABET-accredited institution, which seems to feature prominently in proposals from IEEE and others. Ideally I think there should be some way around that, but alternatives I'm familiar with aren't great either (e.g. FINRA).

[malandrew]

> experts

Being an expert means you know your tools well. If you can't code well and coding is a critical part of your toolset, then you're not an expert yet. You don't get a free pass because you allegedly are strong in other parts of your craft, especially if the part of your craft you're weak in can be this problematic.