[the_watcher]

Thanks for sharing the second part! I’ve worked places as a DS where we were ramping up A/B tests, and it’s honestly one of the worst experiences a DS can have - everybody is asking why you need such large samples, then when you get null results, everyone wants an explanation for why (“this idea didn’t do what you wanted it to” isn’t an acceptable answer), etc.

You’re entirely right that running a bunch of tests is a really effective way to advocate for better resources to run the tests. While I’ve never worked anywhere that ran out of ideas to game the testing incentives, it’s definitely true that people who initially fall into the trap of gaming the tests come around with some experience and in general the org builds better intuitions and culture around how to use tests.

It may be that an org that _has_ a culture of testing just has them constantly running in the background but they’re minor in terms of time spent running them, but that building a culture of testing initially involves the whole org focusing on it to a seemingly ridiculous extent, as it’s the only way to generate enough momentum to get the infrastructure and institutional knowledge right.

[pyrolistical]

What is DS?

[Thorondor]

Data scientist, I'd guess.