[wting]

One of the ways to measure long term impact is through the use of a golden cohort that is never opted into experiments. Unsurprisingly, I could not get this work prioritized on the roadmap.

We also worked with growth consultants (read: Bay Area B2C product leads) in scoping out some of these ideas. We accrued what I call "product debt" where we launch the MVP but never followed up to polish the feature[0] as they don't improve KPIs.

I assume this is the same with Growth teams everywhere but am happy to be corrected.

Regarding long term impact, we measured this through various dimensions in marketing, recruiting, and user research. The outcomes are largely positive.

______

0: One feature I argued for was an opt out of the mobile app interstitial. It makes sense to show it once or twice, but users aren't going to download the app just because they saw it 50x.

[Ozzie_osman]

Yeah, golden cohorts can work, but they are really hard to pull off, especially for logged out traffic (which is where you'd use most of these patterns anyway). Good luck tracking me over 6-24 months across different devices and locations. And cross-contamination is hard to prevent (for instance, the golden cohort might suffer from global effects like worse content due to loss of power users or even from stumbling across brand-damaging threads like this). It also just adds a lot of product complexity to keep behavior around that long.

That said, they can work. Twitter famously did something like that for their time-based vs algorithmic feed and I think YouTube does it pretty regularly.

The biggest issue, though, is that by the time you get results from any long-term experiments, most of the decision-makers (PMs, EMs, etc) have probably moved on away having taken credit for the short-term wins they delivered.

[wting]

The person who created Twitter's experimentation platform is also at Reddit, and heavily influenced Reddit's experiment design and review process.

But yes, a revolving door of product leaders and decisions is going to bias towards short term optimization.

[nwienert]

It’s shocking to me how people sell out like this. You have to know deep down that all these hostile short term juicers destroy the brand, each malfeasance creating more room for a competitor. I mean you guys replaced Digg, cmon.

The audacity to claim “it works”, in italics no less.

The real shame of the current tech companies is they have no principles, no long term vision. They all feel like they follow the same curve, a bunch of managers hitting KPIs during their 2-5 year stint before trading up, ending in some PE firm diving in at the end for the final squeeze.

They’re lemons being juiced dry, when they should be a garden of lemon trees.

“But we got 20% more juice than last year!!”

Yea, you did.

[MomoXenosaga]

It is not shocking when you realize it is about self-interest. You get your bonus and salary in the short term who really gives a shit what happens 7 years from now?

[nwienert]

Not shocking for the managers, just sad. Shocking that executives don’t get it so consistently.

[Flow]

> Regarding long term impact, we measured this through various dimensions in marketing, recruiting, and user research. The outcomes are largely positive.

You almost make it sound like this is something the users like and want.

As a long time Reddit user I can say that if they force the new site on to me along with their mobile app, I'm gone. It's not worth it.