[shadowsun7]

This is an interesting question! The answer is to hold conclusions loosely and test; always test.

One way to pick who to pay attention to is ‘believability’ — meaning that you listen to those with at least 3 successes, and a coherent explanation when probed.

Pointing out survivorship bias is a common rejoinder to this view. But when you’re trying to put things to practice (not get at some universal truth like a researcher would) you often cannot wait for perfect samples. So you pick certain practices from believable people and test them against reality, and then hold the lessons loosely, making sure to update based on further experiments (which is necessary because life is messy and full of confounded variables).

Over time, it becomes clearer what is useful and works for you, and what isn’t and is perhaps a quirk of the other organisation’s context. But I’m not saying anything new; this is how we learn from life.

Related: Brian Lui’s loose feedback loops https://brianlui.dog/2020/05/10/beware-of-tight-feedback-loo...

And the problems of learning from experience: https://commoncog.com/blog/the-hard-thing-about-learning-fro...