[rontoes]

This is an incredibly poor take. It's well known that academic software doesn't follow great software engineering practices. The model is an incredibly complex piece of software that attempts to model stochastic behaviour. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model.

>On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector.

This is another level of crazy.

[vikinghckr]

>This is an incredibly poor take.

No it isn't.

> It's well known that academic software doesn't follow great software engineering practices.

That doesn't make it acceptable. Also, it's also "well known" that vast majority of academic work has zero tangible effect on society. This isn't one of those works. It's possibly the most important piece of academic work that has happened in recent memory. So the bar for this is MUCH higher than typical academic research work.

>. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model.

It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies.

> This is another level of crazy.

No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.

[spamizbad]

> No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.

The insurance industry is expected to ask the government for bailouts because none of their models can account for the fallout from this, just like AIG did during the '08 crisis.

[twelve40]

Not to protect the insurance industry, which I know nothing about, but how does their hypothetical failure, misery and bailout relate to, or validate the study in question?

[arkades]

Criticism: Academia fails to model the real world, in contrast to industry ("is way behind industry").

Response: An entire industry of competing and well-funded actors that specialize in modeling and predicting expensive failure states also failed to model the world accurately.

Implication of Response: Where is the evidence that this is inferior to industry?

[no-s]

>>Implication of Response: Where is the evidence that this is inferior to industry?

A meta implication is that expert models were insufficient to inform decisions.

[smsm42]

Well, even if they did they'd ask for it because they know they'd likely get it. And then what exactly is the incentive to account for it and pay for it with your own money if you can not account for it and get money from the government if anything bad happens?

[Nimitz14]

> It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies.

Wrong. Nobody decides policy based on whether there will be 320k or 400k deaths. What matters is the order of magnitude.

[thu2111]

80,000 deaths is more than double the current number of deaths in the UK, which is itself very high.

This difference alone is the difference between deploying the army to build field hospitals and emergency seizures of industry to make it happen, or doing nothing. And it comes from floating point differences between AVX and non-AVX hardware. The apologetics for this on HN are absurd.

[Nimitz14]

> 80,000 deaths is more than double the current number of deaths in the UK, which is itself very high.

That is not relevant.

> This difference alone is the difference between deploying the army to build field hospitals and emergency seizures of industry to make it happen, or doing nothing. And it comes from floating point differences between AVX and non-AVX hardware.

No, 320k or 400k deaths will not make the difference to decide whether the army will be deployed.

If the 80k were the difference between 100 and 80100, then it obviously would impact policy.

What matters is the relative difference, not the absolute. How is this so hard for you to understand?

[smsm42]

There's a big difference between randomized model and code giving random results because you have race conditions in the code. If you do an experiment, you can expect some randomness because no experiment gives same value ever. But behind it is the common reality that you can measure. Now, if during an experiment somebody would come and kick the equipment, spit on it, spill water on it, subject it to random electric shocks, shoot at it, etc. and then when results come out all over the place say "well, it's supposed to be random!", completely ignoring what was done to it - you'd say that person is insane, it's supposed to be random but not like that.

If you code has race conditions that make your simulation roll out and produce different results, then you aren't measuring the result of the simulation, you are measuring race conditions.

Of course, you could be lucky and race condition could be harmless or contribute almost nothing to the result. But this is unusual, needs proof and indications are it's not the case. Responding to that "well, it's stochastic so what do you want!" is not the way it works.

Poor code quality is a sad reality in many scientific projects, yes. But it's not just a code that is unreadable. It's the code where race conditions knowingly mess with the model results! You can't just dismiss that.

[below43]

Suggesting all academic epidemiology be defunded really highlights the true intentions and bias of the writer.

It might be better, perhaps, for an academic to perform a code review instead.

[lbeltrame]

Regardless of the bias of the piece, AFAICT the GH issues linked to weren't made by the author (to be honest, my own bias suggested, before reading the text, that the author was the only one that pointed out the mistakes) but by others trying to use the model.

This makes the criticism (although partisan) at least worthy of attention.

[ilyaeck]

Big decisions impacting many livelihoods were made based on this model's projections. Its scope is way beyond academic.

[hartator]

> It's well known that academic software doesn't follow great software engineering practices.

Well, know that literally trillions of dollars have been lost, it's not a surprise that the computer model at the origin of the panic is being scrutinized.

[s9w]

Why do you think that's a crazy conclusion? That code is worthless - such quality of work would not be accepted in the private sector. And it's doubtful it can be of any value at all.

So when the work they do is this bad, I think it's reasonable to question whether they should be continued to be paid for no value to the public.

[lbeltrame]

> such quality of work would not be accepted in the private sector.

To be fair, if such code would be used for a paper and the flaws came out, I think it would warrant a retraction.

I think I saw not too long ago a story right here where a software error caused completely wrong results.