[mcfunley]

Author here. The main thing that inspired this happened a few years before I wrote it down. Etsy had gotten a new CEO, and they spent one of their first few weeks in long hours at my desk, iterating on the homepage design in what could only be described as a radically fast iteration loop. We'd ship a tweak, look at statsd for ten minutes, then change something else. This would have been a bad idea for all of the reasons of statistical validity listed, even if we hadn't built statsd to use UDP.

Emphasizing working on the homepage was also analytically dumb in a more meta way, since item/shop/search were really nearly all of traffic and sales back then. Anyway, I felt motivated to get that person to think first and fire the code missiles second.

At the end of the day, I think back on it fondly even though it was ludicrous. Shipping that much stuff to production that quickly and that safely was a real high water mark in my engineering career and I've been chasing the high ever since.

[piker]

Isn’t that last sentence sort of a reason to prefer real-time analytics? If you can make development a fast paced game, no doubt you’ll keep your team more productive and engaged. Granted, it needs to be engineered in a way to ensure that productivity is aimed correctly (“how we decide which things we do”) as you point out in your great article.

[teruakohatu]

There is a good chance the OP shipped changes that would have positively impacted the bottom line, but after 10 minutes of real time analytics it was replaced with something else because it performed poorly in a single 10 minute period.

You can ship A/B tests quickly and many websites do, but decisions are made after a statistically relevant time period.

[tunaoftheland]

Good question, though what you have in mind might be real time metrics, not analytics. Even then you might not need real time metrics to know whether your rapid changes are breaking things. An already established dev culture built on CI/CD, actionable health checks, feature flags/toggles, easy release rollbacks in emergencies are what you’d want. This way, your deploys are boring and you can focus on introducing new regressions, uh I mean features, fearlessly. :)

[mcfunley]

No, it’s orthogonal to the analytics