[ZephyrBlu]

Feature flags need to be treated as short-lived and experimental otherwise they end up getting abused for everything and make it very difficult to reason about your application.

If it's config/customization, it should be in code. If it's experimental it can be a flag until it solidifies, and then it needs to get moved to code.

When I was at Shopify a couple of years ago they mandated that feature flags had to be short-lived (Like 2-4w lifetime tops, some had exceptions) because they would end up getting left in code and never cleaned up, or for extended periods of time like months. Hard to tell if it's genuinely a "feature flag" or actually just a normal part of the system at that point.

Feature flags being flipped in prod was also a major source of incidents, in part because people didn't treat them as experimental and with the associated risk profile of something experimental.

The only exception where having long-lived flags was useful and required was for operational killswitches (E.g. disable Apple Pay because it's having issues), but that is explicitly not application config.

[WhitneyLand]

Agreed.

This is the kind of design wisdom that’s both true and difficult to win an argument over.

It reminds me of arguments related to over-engineering and complexity. The principles are super important to having a codebase that scales and continues to be efficient to work in as the team grows, but they are hard to objectively measure.

Locally or in isolation something may sound like a great idea. Being able to step back and see the greater ripple effects require some experience and intuition that can’t always be used to convince people otherwise.

[baq]

I disagree with just about everything you said being a problem except the process of cleaning up is absolutely required.

Notably feature flags triggering incidents is expected and desired vs the alternative of shipping the code and having to roll a release back because there is no other way to remove the feature from prod.

[ZephyrBlu]

In a company the size of Shopify people flipping their feature flags would very often impact *other teams*, and like I said feature flags got abused with even seemingly innocuous changes being put behind them or being left long periods of time before being fully used.

When someone else flips a flag that impacts your team and they have no idea they even caused a problem, it becomes very difficult to resolve the issue. Usually you can check for recent deploys, instead you have to go and guess at which feature flag which was recently flipped could possibly be affecting your code. I experienced this several times.

Also, it was actually more desirable for most of these things to go straight to production. Test it properly before shipping, then when you ship it soaks on a 5% traffic canary at which point you can monitor and cancel the deploy if you see errors. That is generally safer than a feature flag rollout unless you are doing something very high impact/risk, in large part because it gives any other team affected by your rollout the ability to respond and be able to easily find the source of errors.

In my org it was a fairly common failure mode to ship something and accidentally cause an issue for another team. Usually it was other teams/orgs shipping things that impacted us.

[echelon]

Runtime evaluated feature flags can always be used for control plane levers and emergency handbrakes.

You just have to label them as such and prevent other teams from fiddling with them.

This is not an antipattern, it's just semantic hand-wringing.

My team managed critical systems in the online flow of billions of dollars of daily payment volume. We also wrote the feature flag system that the rest of the company used. Not only were we completely fine with feature flags as long-lived control plane levers, we heavily used the system that way ourselves.

You just have to clearly distinguish between ephemeral rollout flags (and clean them up or expire them) and the permanent control plane levers.

It's the exact same functionality for both sets of tools. Just different practices around the two usages.

[ZephyrBlu]

I completely agree with your distinction and that is exactly what they mandated :)

I don't think that is what most people colloquially mean by "feature flags" though. Even most teams in Shopify abused "ephemeral" flags for long periods of time.

When they rolled out the mandate it was very annoying for my team because we had a lot of operational flags like you're describing that we needed to get exemptions for.