[brookst]

The problem is you often don’t know what metrics are meaningful until after the fact.

Like, if new order starts drop by 4% while traffic remains constant, what happened? If that happens, you might want to see if people are using the menu more because something got harder to find on the page.

I doubt anyone is looking purely at menu opens as a metric, unless maybe trying to reduce it. But for ongoing funnel and ad hoc investigations it could be useful. So you collect it.

The obvious answer is sampling rather than collecting for every user, but then you get into complicated statistics about required sample sizes if you want to correlate multiple actions across tech and demographics. Again, easier to just collect it all.

[6510]

Do many oversized tests but carefully avoid testing the test themselves because why would you? Would you test the test of the test too? etc? Someone might one day conclude the tests cost more than they are worth but it wouldn't be the ones hired to inflict them.