[pdonis]

> Each experiment is not a result per se but a sample from a distribution.

But what distribution? What is this "distribution" that we are taking a sample from?

The frequentist says: because the two experimenters have different intentions, the experiments they ran are samples from different distributions.

But the Bayesian says: the experimenter's intentions can't affect things like how dice rolls come out or how well a given treatment works on a given patient. The actual "distribution" is the set of all factors that do affect how the dice rolls come out or how well the treatment works on each patient. And those factors are the same for both experimenters; their different intentions don't affect that. So both sets of data are samples from the same distribution, not different ones.

> How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have taken.

If you're going to state it this way, then the Bayesian response is: "all courses your experiment could have taken" has nothing to do with the experimenter's intentions. The experimenters can't magically make the physical world and the biology of humans work differently depending on what stopping criterion they choose. And the physical world and the biology of humans is what determines "the courses your experiment could have taken".

In other words, when the frequentist makes up "distributions" based on the experimenter's stopping criterion, they are, whether they admit it (or even realize it) or not, making a claim about how the physical world and the biology of humans works that is obviously false.

[AbrahamParangi]

This seems to assume that intentions "don't count" in some way, as if they were nonphysical, whereas unless you presume a supernatural soul, they are as physical as any other part of the experiment.

[pdonis]

> This seems to assume that intentions "don't count" in some way, as if they were nonphysical

Not nonphysical: just not part of the physical degrees of freedom that can affect things like how die rolls come up or how well a given treatment works on a patient.

The experimenter's intentions (not about the stopping criterion, but about other things) can of course be upstream physical causes, so to speak, of things like what the actual process of the treatment is, and that can, of course affect how well the treatment works. But in the scenario under discussion, all those things are stipulated to be the same in both experiments. And once that is specified, whatever physical variation corresponds to the variation in the experimenters' intentions cannot affect the results.

[psychoslave]

> Not nonphysical: just not part of the physical degrees of freedom that can affect things like how die rolls come up or how well a given treatment works on a patient.

For a dice that is not a concern (unless animism is taken into consideration), but when humans are on both side of the equation, how do you get rid of all the social and psychological effects that imply, including placebo and the desire to see the study bend in some direction, be it at some unconscious level?

[pdonis]

> when humans are on both side of the equation, how do you get rid of all the social and psychological effects that imply, including placebo and the desire to see the study bend in some direction, be it at some unconscious level?

You don't. But again, in the scenario under discussion, these are stipulated to be the same in both cases. (Or more precisely, the underlying unconscious factors involved have the same distribution in both cases.) So again, these kinds of "intentions" don't make the distributions different in the two cases.

Another comment is relevant here. The whole point of things like double blind studies in medicine is to make it the case that, whatever unconscious factors are involved along the lines you describe, they don't change the underlying distribution from which the sampled results are drawn. In the scenario as described in the article, it was assumed that all of these precautions were taken. That is part of the reason for the article's statement that the experimenters' intentions about the stopping criterion can't affect the results.

Of course if you know that in a particular case, those precautions were not taken, that changes how you view the results. But Bayesian analysis can cover this case too: you just expand your hypothesis space and your prior to include things like "the experimenters unconsciously influenced the results in different ways because of their different stopping criterion". The article excludes this possibility in the scenario it describes, but in the real world, yes, we know it is possible to have study designs that don't eliminate this failure mode, and our analysis should allow for that in cases where the study design was such that it might have happened.

[psychoslave]

> But again, in the scenario under discussion, these are stipulated to be the same in both cases.

That's not what I read. The two studies imply that they are conducted under two very different mindsets, which will most likely also mean people will receive a different human treatment. At this point, the statistics you will get out of it is almost ornamental. Sometime the most significant information to extract from a description is not the one that is the most obvious that is pointed at as the thing you can quantify and draw comparisons.

[pdonis]

> The two studies imply that they are conducted under two very different mindsets, which will most likely also mean people will receive a different human treatment.

Even if the studies are conducted using the standard double blind protocol for medical and social science research, where the experimenters who have the different stopping criteria are not involved in any of the actual experimental activities? They don't do the random assignment of patients to treatment and control groups; they don't administer any of the actual treatments or placebos; they don't interact with the patients or the treatment personnel in any way during the experiment; and they don't inform anyone who is actually involved in the experimental activities of their intentions, regarding stopping criterion or anything else.

You're saying that even if all this is done--and, as I've said, this is standard procedure in medical and social science research--it is still impossible to prevent the experimenters' intentions--which aren't known to anyone else involved--from affecting the results? Remember that, as I've already said, the whole point of these double blind protocols is to prevent the experimenters' intentions from affecting the results. You're saying that's a fool's errand?