[6gvONxR4sf7o]

I only skimmed it, so forgive me if I got this wrong. The causal model used here makes some incredibly strong (unlikely to be close enough to accurate) assumptions. Are these results valid if there are unobserved confounders or selection bias?

[benmaraschino]

Well, at the end of the day, you can never really be sure of strongly ignorable treatment assignment/unconfoundedness, no matter what problem you’re working on. Especially if you’re an economist or an epidemiologist working with data that’ve been collected by someone else—you can’t exactly easily go back and measure more predictors of treatment assignment. But if you’re running a website, there are a lot of variables you can measure on the user end and more opportunities to iterate, and so SITA then begins to look more and more like a better bet.

[6gvONxR4sf7o]

Maybe you run a different kind of website or ask different kinds of questions, but despite being able to measure all kinds of things, there's so much at my job that you need experimentation for. You do the observational study, and it points in this direction. Sometimes it's true and sometimes it's not. Selling this kind of observational analysis as 'you don't need A/B tests anymore' is totally disingenuous.

[otterk10]

Thanks for the feedback! I totally agree about observational studies being suggestive but not replacing A/B tests - that’s why the main use case I listed in the blog (and how current customers have used the product so far) is “prioritization of a/b tests”, not replacing a/b tests themselves. The language around “simulating a/b tests” is just a way to try to concisely explain to someone at a high level who may not be very technical or has never heard of an observational study. Happy for suggestions on how to better explain observational studies to less technical customers without over-selling! It’s something we’ve been iterating on ourselves.

[lern_too_spel]

You're right. I would go so far as to say the assumptions that their model makes are never valid in practice on real world problems.