[epcoa]

> complex systems where lots of data exists

Not a lot of high quality data exists for human health. Clinical guidelines for many diseases are built around surprisingly scant evidence many times.

> even if they're so complex no human can understand them?

That’ll be wonderful to explain in court when they figure out it was just data smuggling or whatever other bias.

[epistasis]

In cancer there's an abundance of clinical trials with high quality data, but it is all very complex in terms of encoding what the clinical trial actually encoded.

Go to a clinical cancer conference and you will see the grim reality of 10,000s of people contributing to the knowledge discovery process with their cancer care. There is an inverse relationship between the number of people in a trial and the amount of risk that goes into that trial, but it is still a massive amount of data that needs to be codified into some sensible system, and it's hard enough for a person to do it.

> That’ll be wonderful to explain in court when they figure out it was just data smuggling or whatever other bias.

What do you mean by this? I'm not aware of any data smuggling that has ever happened in a clinical trial. The "bias" is that any research hypothesis comes from the fundamentally biased position of "I think the data is telling me this" but I've seen very little bias of truly bad hypotheses in cancer research like those that have dominated, say Alzheimer's research. Any research malfeasance should be prosecuted to the fullest, but I don't think cancer research has much of it. This was a huge scandal, but I don't think it pointed to much in the way of bad research in the end:

https://www.propublica.org/article/doctor-jose-baselga-cance...

[epcoa]

By smuggling and bias I meant in an ML model. Smuggling was a bit informal, but referring to models overfit on unintended features or artifacts.

[londons_explore]

but we have well established ways to deal with those... test/validation sets, n-fold validation, etc.

Even if there was some overfitting or data contamination that was undetected, the result would most probably still be better than a hand-made decision tree over the same data...

[wizzwizz4]

Hand-made decision trees are open to inspection, comprehension, and adaption. There is no way to adapt an opaque ML model to new findings / an experimental treatment except by producing a new model.

[epcoa]

Ok, until you can sue the AI you need to find a doctor ok putting their license behind saying “I have no idea how this shiny thing works”. There are indeed some that will, but not a consensus.

[londons_explore]

Great. Lets have the few that are willing to put their license behind that do so, and then a study showing that those people get better cure rates than those who do not...

And then the decision tree can be rewritten as "Do ${New_method}".