[klenwell]

The conclusions from the first critical code review cited:

All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.

https://lockdownsceptics.org/code-review-of-fergusons-model/

[pjc50]

That's the most biased opinion imaginable by an astroturf group. There's no information on who they are, but the article currently on their front page is by Toby Young, who is a Spectator/Quillete guy and to be found backing almost all stupid ideas within British politics. https://www.spectator.co.uk/article/This-lockdown-may-kill-m...

[rscho]

Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health?

What could go wrong?

[SpicyLemonZest]

If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.

[rscho]

That's also true. Now the million $ question:

Whom would you like our society to rely on to generate quality work?

I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions.

There is no smart solution to a stupid problem. But the truth is that _as an institution_ the NHS is the only actor whose mission is to optimize towards public health.

[SpicyLemonZest]

In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good results is profitable!

[mturmon]

> In most fields, our society relies on private industry to generate quality work ...

This is just not true. Military, police, courts? What does "most fields" even mean?

[SpicyLemonZest]

My job, food, apartment, utilities, entertainment, all come from private companies.

[klyrs]

> it seems only fair to say that academics optimize for prestige and publication in good journals.

On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time.

On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles on the basis of their veracity (or, if incorrect, as punching bags). Of course that's a fallible process and just as vulnerable to cronyism, but when researchers are caught cooking the books they're discredited, not rewarded.

[friday99]

Authors aren't actually incentivized to find accurate results, but rather publishable results, which typically means novel. Researchers also cite articles based on their impact, not their veracity. There are plenty of instances of retracted results continuing to be cited as if they are still accurate.

There are issues with both industrial and academic research, but I do think that industrial research is more transparent in its motivations.

[toolz]

Everyone working for a living optimizes their work towards making more money. That doesn't change for people funded with public money. It's pretty widely accepted that the people in charge of public funding (politicians) sometimes act outside of the publics interest for self-gain.

I make no claim as to which achieves better results for the public because it's such a complicated problem, but I think it's rather naive to just assume publicly funded incentives are more aligned with social health than private incentives.

[spamizbad]

That last part is mindblowing given the fact that the health insurance industry in the US is trying to argue it shouldn’t have to pay for COVID-19 treatments because it’s part of a pandemic and not part of normal medical treatments.

I guess if your model is:

   if pandemic and COVID:
       is_covered = False

You can have a very clean pandemic model.

[lern_too_spel]

That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones.

The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.

[Joeboy]

> That was a shitty code review

Damn straight. The only sense in which it was not a shitty code review, is that it didn't actually review any code. Looking into the linked tickets is a time consuming faff, but people should actually do it before taking the blog post at face value.

[klenwell]

I agree. Moreover, the cited bug ("predictions varied by around 80,000 deaths after 80 days") doesn't really seem to impact the over all policy implications: no lockdown means exceptional number (400 to 480k) of deaths.

Unless the entire simulation is bogus, it comes off as nitpicking.

[thu2111]

Seriously?

Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct.

Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?

[lern_too_spel]

What does reading uninitialized memory or reusing variables that weren't set to zero have to do with seeding issues? Read my comment again and reply to its content instead of making up a comment that you would like to reply to.

[Nursie]

Yeah that's a weird set of conclusions there. I would think the papers based on it want another look though.