[gwern]

It's a good comparison. An effect can be real, and people just not notice it, ever. Let's round off the ad effect to 10% loss of users. How do you notice that? You can simulate out traffic, and decrease the mean 10% at some random point and draw time-series: it's actually quite hard to see, particularly with any kind of momentum or autocorrelation trends. And that's with website traffic where you can quantify it. How do you measure the impact on something more intangible? If people can not notice that effect of ads, they can certainly not notice subtler effects like the long-term harm of casually breaking links...

Is <10% worth caring about? It's certainly not the difference between life and death for almost everyone; no successful business or website is going to shut down because they had ads, or because they broke old URLs. On the other hand, <10% is hardly trivial, people work hard for things that gain a lot less than that, and really, is defining redirects that hard?

[sbierwagen]

Speaking of noticing small effects, Mechwarrior Online was four months into its open beta before anyone noticed that an expensive mech upgrade that was supposed to make your weapons fire 5% faster actually made them fire 5% slower. http://www.howtospotapsychopath.com/2013/02/20/competitively...

[parafactual]

This principle seems like it extends far beyond site traffic, especially since something like life satisfaction is much harder to measure.

[gwern]

Yes, it's a kind of slippery slope: as typically set up, changes are biased towards degrading quality. If you run a normal significance test and your null is leaving it unchanged, then you will only ever ratchet downwards in quality: you either leave it unchanged, or you have a possibly erroneous choice which trades off quality for degradation, and your errors will accumulate over many tests in a sorites. I discuss this in the footnote about Schlitz - to stop this, you need a built-in bias towards quality, to neutralize the bias of your procedures, or to explicitly consider the long-term costs of mistakes and to also test quality improvements as well. (Then you will be the same on average, only taking tradeoffs as they genuinely pay for themselves, and potentially increasing quality rather than degrading it.)