[6gvONxR4sf7o]

Maybe you run a different kind of website or ask different kinds of questions, but despite being able to measure all kinds of things, there's so much at my job that you need experimentation for. You do the observational study, and it points in this direction. Sometimes it's true and sometimes it's not. Selling this kind of observational analysis as 'you don't need A/B tests anymore' is totally disingenuous.

[otterk10]

Thanks for the feedback! I totally agree about observational studies being suggestive but not replacing A/B tests - that’s why the main use case I listed in the blog (and how current customers have used the product so far) is “prioritization of a/b tests”, not replacing a/b tests themselves. The language around “simulating a/b tests” is just a way to try to concisely explain to someone at a high level who may not be very technical or has never heard of an observational study. Happy for suggestions on how to better explain observational studies to less technical customers without over-selling! It’s something we’ve been iterating on ourselves.