[Dylan16807]

So you have a cancer that normally has a 90% fatality rate after a year. Someone gives you evidence that they gave this drug to everyone with this cancer in their hospital, and 70/73 survived for a year. Not properly randomized, not properly controlled. Would you really reject their data and find it unconvincing?

[gus_massa]

If the results are so good it would be impressive. But it will be necessary to take a look at the data. Dollar to doughnuts they have a horrible methodological mistake, like a bad classification of the patients, or using a weird definition of cure [1], or cherrypicking the patients that get cured.

[1] We have a big announcement of a miracle cure in Argentina in 1986. It was crotoxina [links bellow] that is a part of the venom of some snakes. One of the problems was that they were comparing CT from different angles and finding fake reductions in the size of the tumors. (Other parts of the study were just frauds.)

https://translate.google.com/translate?sl=auto&tl=en&u=https...

https://translate.google.com/translate?hl=&sl=es&tl=en&u=htt...

https://translate.google.com/translate?hl=&sl=es&tl=en&u=htt...

[Dylan16807]

Valid issues, but you can have all of those problems in a randomized controlled trial too. That's not a very compelling argument that the non-random poorly-controlled version is any worse in this hypothetical.

[gus_massa]

For example if the "usual" death rate is 90% and you select some group of patients for security reasons, like 18<=age<=60, perhaps the death rate is reduced to 80%. If the testing and control group is randomized, then if there is no effect of the drug you will get the same 80% in both groups. If the group is not randomized, you can't be sure that any of the selection criteria is the cause of the improvement.

[Dylan16807]

You can look at the selection criteria and perform an analysis. If the data is clear enough, you can still pull a signal out of the noise. I assume that's the reason you took a 70/73 survival rate in the hypothetical I posed and reduced it to 20% in your version. And to be clear, in my version it's a general hospital and they give the treatment to everyone that has this diagnosis. There are no intentional selection criteria, it's mostly just who lives in the area.

[gus_massa]

[Sorry for the late reply.]

> If the data is clear enough, you can still pull a signal out of the noise.

It is theoretically possible, but very difficult. Unless you use a lot of people in the trial, but then you must ensure that the measurements are done in a consistent way.

> in my version it's a general hospital and they give the treatment to everyone that has this diagnosis

Does it include pregnant women and babies with less than 1 year? Does it include people with more than 90 years? Does it include people that goes to the hospital in an ambulance because they are almost dying, like a hearth attack? Does it include someone that had one of the lungs removed and is under a chemotherapy treatment?

What about people that can't sign the form for the experimental treatment? Just signing a form is a selection of people that is not toooooooooooooooooooo bad.

What about asymptomatic people? Does your are has the same policy to test everyone/someone/noone than the region you are comparing with? What about the effects of temperature or humidity?

What about diet? Poor people may have a bad diet, with a low amount of vitamins and that can affect the illness. Some countries drink a lot of milk and some very few, some countries add vitamins to the milk.

What about the median income? If the city has a few hospitals, there will be one closer to the poor area and other closer to the rich one. Some people has health plan that include one hospital(s) but not other hospital(s). How does it affect the selection of people in the hospital? Different countries have a different definition of poor.

Some hospital are famous and get more of the strange/difficult cases after the standard hospitals give up or realize it is a complex case.

I may be missing other factor, or overestimating some of them, but it is very difficult to be sure that you know all the things that change the cure rate and that you can correct the result.

[Fjolsvith]

In this case, it is a bit easier to deduce a successful result, in that the patient survives death versus having a tumor shrink in size.