[wduquette]

"they are AWFUL at stats."

SF author Michael Flynn was a process control engineer as his day job; he wrote about how designing statistically valid experiments is incredibly difficult, and the potential for fooling yourself is high, even when you really do know what you are doing and you have nearly perfect control over the measurement setup.

And on top of it you're trying to measure the behavior of people not widgets; and people change their behavior based on the context and what they think you're measuring.

There was a lab set up to do "experimental economics" at Caltech back in the late 80's/early 90's. Trouble is, people make different economic decisions when they are working with play money rather than real money.

[Projectiboga]

Expermential Design is one of the big four adacemic subjects within Statistics. The math is complex even before the issues of the effects of the expermential situation.

[SkyMarshal]

Oblig link to Norvig's "Warning Signs in Experimental Design": https://www.norvig.com/experiment-design.html

[dgfitz]

> Trouble is, people make different economic decisions when they are working with play money rather than real money.

Understated even. Ever play poker with just chips and no money behind them? Nobody cares, there is no value to the plastic coins.