[zwaps]

Maybe I am too cynic but what we are really talking about here is causal inference for observational data based on more or less structural statistical models.

Any researcher will tell you: this is really hard. It is more than an engineering problem. You need to know not only how to deal with problems, but rather what problems may arise and what you can actually identify. Most importantly, you need to figure out what you can not identify.

There are, at least here in academia, only a limited set of people who are really good at this.

Long story short: even if offline analysis is viable, I doubt every team had the right people for it, making it potentially not worthwhile.

It is infinitely easier to produce a statistical analysis that looks good but isn’t, than one that is good. An overwhelming amount of useless offline models would, statistically speaking, be expected ;)