[ulf-77723]

I worked in a SaaS company selling an A/B Testing app where I saw a lot of tests and their outcomes from different clients.

Of course there are the low hanging fruits, which scored the clear success. Most of the tests although were not a clear success, but just a slightly better conversion rate.

Multiple tests applied after each other slowly increased the overall value. If taken the advice not to implement the changes of the small success, the overall amount would not have changed.

[abakker]

Indeed. It's almost like the original article is clickbait :) The problem is that we have a term for what he's describing already - diminishing returns. For some of these "mediocre successes" you may find that your business is already down the optimization curve on some choices, but, having done a test and found a signal to improve, why not?

After all, pretty much all of writing, music, product design, prototyping, etc. is iterative. It's a purely investor mindset that would have you think that small optimizations do not pay back. Having that mindset in creating something is a great way to give up after one draft of a good idea.

[regularfry]

"Just slightly better", if you can have confidence that it's not a mirage, is a success and should be taken as such. I don't think that's inconsistent with the article, which is more about the danger of allowing a gap between success and failure which, if you land in it, leaves you with no clear direction to go in.

[mjmahone17]

So long as only 10% or less of your incremental successes are mirages, and so long as the downside of shipping a mirage is only a small incremental harm, then shipping 9 success and 1 harm should still get you an overall ~78% win vs just shipping the 9 successes (assuming the bad result is an equivalently negative result to the good ones).

How much are you willing to spend to reduce the downside risk, and how many “good” experiments are you willing to throw away in the process?