[muraiki]

There are other benefits to Bayesian data analysis besides being able to handle limited data. There are problems with the outputs of frequentist analysis around the quantification of uncertainty. For instance, from simulation studies we know that the aleatoric coverage probability for confidence intervals of a selected confidence level varies depending on the size of the difference in plausibility between the null and alternative hypotheses. And a given confidence interval says nothing about epistemic uncertainty for this particular experiment. This can make the outputs of frequentist analysis difficult for stakeholders to utilize, whereas Bayesian epistemic probabilities are generally more easily understand by stakeholders, and can directly feed quantitative decision analysis methods.

A good introduction to some additional problems with frequentist methods vs Bayesian and likelihoodist methods is this: https://gandenberger.org/2014/08/26/intro-to-statistical-met...

An interesting book on adapting frequentist methods to create confidence distributions that can better express uncertainty and can optionally incorporate prior information using likelihood functions is this: https://www.cambridge.org/core/books/confidence-likelihood-p...

[wenc]

I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ). I’m sure there are fields where such statistical rigor is called for (where the quality of the data is so high and accurate that the variation is in the analysis — often the case with machine data).

For everything else there’s so much error. Being to quantify uncertainty is great — it’s a signal we need to collect more and better data. But so often we have to move ahead with uncertain data.

Interestingly, in business, taking action (even if wrong) produces outcomes that are much better signals to learn from than having statistically rigorous analyses, so many times there’s a bias for action rather than obsession over analysis.

But of course in some fields being wrong is costly (like clinical trials) so I can see UQ being more useful and prominent there.

[srean]

In many of my projects I have had to incorporate the knowledge/intuition of domain experts. New product launch (by self or competitor), some unseen change in the operating environment. These are event history of the 'does not repeat but rhymes' variety.

Although there may be no data collected from the time something similar happened before in history, experts can reason through the situation to guesstimate the direction and magnitude of the effect in qualitative terms.

Bayesian formulations are very handy in such situations.

>I’ve found that in many applications, the difference between a frequentist analysis and a Bayesian one is unlikely to make a difference in the decision making (even with UQ).

In that case you may find the following interesting

https://en.wikipedia.org/wiki/Lindley%27s_paradox

"Lindley's paradox is a counterintuitive situation in statistics in which the Bayesian and frequentist approaches to a hypothesis testing problem give different results for certain choices of the prior distribution."

How likely is Lindley's Paradox likely to show up in practice ? well there is Bayes for that (tongue firmly in cheek).

[wenc]

Unless it’s a stacked analysis when results of one study depends on another, usually experts just eyeball the frequentist results and take a judgment call — that’s been my experience (not generalizing but I think in business people like doing things that are simple and easy to understand).

I think it’s definitely possible that Bayesian and Frequentist approaches give different conclusions but in practice it doesn’t alter the final decision. Analyses guide decision making but in the end decisions are made on consensus, narrative and intuition. Statistics is only the handmaiden rather than the arbiter.

[srean]

> usually experts just eyeball the frequentist results and take a judgment call

Indeed, but that does not make it right or rational. Bayesian method helps keep things rational. This is pertinent because human brains are terrible at conditional probabilities.

One can always argue that data analysis is usually just window dressing and decision making is mostly political and social. Empirically you would be mostly right if you take that position. One cannot argue against that factual observation.

The more interesting question is, if the decision makers aspire to be rational, which method should they use. I have used frequentist and Bayesian methods both. I made the choice on the basis of the question that needed answering.

For example, when we needed to monitor (and alert on) a time varying probability of error (under time varying sample sizes) -- Bayesian method was a more natural fit than say confidence intervals or hypothesis tests. Bayesian methods directly address the question "What is the probability that error probability is below the threshold now, considering domain expert's opinion about how often it goes below the threshold and how the data has looked in the recent past?"

[wenc]

> Indeed, but that does not make it right or rational. Bayesian method helps keep things rational. This is pertinent because human brains are terrible at conditional probabilities.

I agree with you on Bayesian methods keeping things rational (consistent within a probabilistic framework).

I would say there are different kinds of rationality however: the 2 that I'm most interested in are epistemic rationality (not being wrong) and instrumental rationality (what works), and in the domain of business (but perhaps not other domains like science and math), we optimize for the latter. This is because not getting analyses wrong (epistemic) is actually less useful than getting workable results (instrumental) even if the analyses are wrong. In fact, some folks at lesswrong tried their hand at doing a startup, applying all the principles of epistemic rationality and avoiding bias, and it did not work out. Business is less about having the right mental model but doing what works. This article expands on this point [1]

The issue is in ill-defined (not just stochastic in a parametric uncertainty sense, but actually ill-defined) domains like business, the map (statistical models) is not the territory (real world) -- it's a very rough proxy for it. Even the expert opinions that Bayesian methods embed as priors -- many of those are subjective priors which are not 100% rational. Not to be cliched but to recycle an old John Tukey saying: "An approximate answer to the right question is worth a great deal more than a precise answer to the wrong question." Frequentist methods are often good enough for discovering the terrain approximately, and in business, there's more value in discovering terrain than in getting the analysis exactly right.

(that said, in these settings Bayesian methods are equally as good too, though their marginal value over frequentist is often not appreciable. One exception might be multilevel regression analysis where you're stacking models.)

[1] https://commoncog.com/putting-mental-models-to-practice-part...

[srean]

> Even the expert opinions that Bayesian methods embed as priors -- many of those are subjective priors which are not 100% rational.

Of course ! Like big bang it needs one initial allowable 'miracle' and does not let irrationality creep in through other back doors.

As I mentioned earlier, I choose the formulation that suits the question that needs answering.

Not sure about the 'what works' vs 'analytic correctness'. How would even one know that something works or have a hunch about what may succeed if they have no mental model to base it upon. Often that is implicit and not sharp enough to be quantitative. Bayesian formulation helps in making some of those implicit assumptions explicit.

Other than that I think we mostly agree. For example both the formulations have a notion of a completely defined sample space, the universe of all possible outcomes. That works in a game of gambling. In business often you do not know this set.

Anyhow, nice talking to you. I enjoyed the conversation.