[sillysaurus3]

Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. Some concrete questions:

- How do you prevent your codebase from becoming if-statement spaghetti?

- How do you prevent new features from being leaked to the user? They'll see the new features in the front end source code. At many companies this isn't an acceptable tradeoff. So do you preprocess the release code and strip out the disabled feature flags? With what?

- What do you use to control feature flags? Just a json file filled with `"foo-bar-feature": true/false`, or something more sophisticated like a control panel that you can use to say "10% of our users will see this feature"?

[hoorayimhelping]

>Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them

I worked at Etsy from early 2012 to late 2015 and I used feature flags every day I was there.

>How do you prevent your codebase from becoming if-statement spaghetti?

You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go.

>How do you prevent new features from being leaked to the user?

You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns.

>What do you use to control feature flags?

You can use a service like LaunchDarkly or Optimizely or you can write your own service, or you can have a file in memory. Usually it starts simple with boolean toggles and evolves into a ramp up system that let's you selectively target users. The important part is that you need to be able to change features without redeploying code.

[sillysaurus3]

You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go.

I don't understand. I was asking about the development source code, not the code delivered to the user. The dev source code isn't stripped of if-statements, right? That's where the feature flag toggling lives. But then it seems like it's easy to devolve into spaghetti.

You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns.

Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this?

I looked up LaunchDarkly. $299/mo for basic team support. Uhh... That's almost the cost of WeWork office space.

Are feature flags really so nascent that there isn't a FOSS solution for it?

[dahart]

> The dev source code isn't stripped of if-statements, right?

When I use a feature flag as an alternative to branching, then I delete the if statements once the feature becomes permananent and gets released to all users. It's analogous to merging a branch. Obviously I'd leave it there during QA and A/B testing, but once toggling is no longer necessary, I remove the toggle.

> Ok, but how? Assume an express server.

You probably have feature flags already and just didn't call them that. But normally in any given project, there are multiple ways to wrap a feature in a toggle.

Here are a couple of examples I use:

- I use handlebars for my frontend templating. When I have a feature that needs to serve or withhold some HTML to/from the user, I put the feature flag in my JSON file of variables that get passed to handlebars. Handlebars has a way to wrap HTML with a boolean test.

- I use the Google Closure compiler to optimize, uglify, and minify my code. When I need a feature flag that is code only, and I don't want users to accidentally get exposed, I use a boolean @define in JavaScript, and wrap feature code in an if statement. When the Boolean is false, Closure compiler removes that block from the output code.

- I set up my own JSON file of config constants that are injected into the code during build. If I have a feature flag to A/B test something, or that I do want released to users but has a secret switch to enable, then I put the constant in the JSON file, and write UI code to toggle the flag. This one usually generates more code than the two above, but whenever I decide that the feature is either dead or permanent, I remove all the code to enable and disable the feature.

- For fully dynamic toggling, I put my boolean config variable into my database. I use Firebase which is a giant pub-sub, so I wrap the toggle in a subscription to the database variable, which might then (for example) inject some HTML and/or toggle the CSS display attribute of some things on my page.

> Are feature flags really so nascent that there isn't a FOSS solution for it?

No, feature flags have always been around, since long before branching. There isn't a FOSS solution for it separate from your framework(s) because how they work depends entirely on your stack. They're also normally really simple, adding the dependencies of another project just to have a boolean in your config file or in your code is usually overkill. That said, there are FOSS and commercial solutions for A/B testing, which is usually a feature flag workflow, but it comes with lots of other stuff you might not need, like the reporting, analytics and statistics parts.

[xorcist]

> I delete the if statements once the feature becomes permananent

Right. In theory. Does this really scale when there's more than a dozen developers touching the same code paths? It's not just you scratching off your own TODOs, it's multiple entagled condiditionals. Can you really delete your flags with confidence that this does not affect any other code path?

Which brings me to my real question: How you you guys test this stuff? When I have experimented with feature flags before this has been to much detriment for testing. Suddenly the new payment API our provider will launch in June needs to be tested with the new backend we'll launching Tuesday, and with the new asynchronous database driver our DBAs wants us to migrate to. And so on. The complexity explodes. Especially when the features have dependencies external to you.

When everything is in master and everything must be in shippable condition you need to test everything with everything else, while previously the new payment API in June could live in it's own branch and test on their own schedule and not bother everyone else everytime the payment provider's test environment took a nosedive for some reason.

So you either end up with brittle tests that can take days to execute, or you end up scratching a lot of the less pressing combinations. Which brings us to the situation above. How can you then be confident to remove conditionals without testing the newly exposed code paths?

It is so far my suspicion that most people who speak of feature flags as an alternative to branches either work with trivial products where most developers fit in a small room (in which you are not the use case for these things) or talk out of some theoretical conviction which has yet to meet with reality. These things are hard, because of all those details which seem small if you squint enough.

[dahart]

To be clear, I personally use branching a lot (e.g., for sprint-sized features) and I create new feature flags rather infrequently (for multi-sprint sized features or for A/B tests), and only for pretty large features. But I can see reasons why some teams/environments do go, or want to go, feature-flag always. Before git, feature toggles were more widespread as a workflow.

As far an entanglement with other code, I would evaluate this wrt branching. If it's entangled, the branching analogy is a merge conflict. The most common case is you can delete the toggle without conflict, and occasionally you have to spend ten or twenty minutes untangling. Every once in a rare while it gets messier.

For testing, the prod tests run the prod config, which is likely with all optional feature toggles turned off. My team would run our own tests with our feature toggles turned on, but not with other teams' feature toggles. I don't run all combinations of features. Usually, a feature toggle will also wrap new tests that belong to that feature.

Again, the same question you ask applies to a branch workflow. Do you run all possible branches at the same time? Is it an exponential combinatorics problem to have multiple branches? Not really, each team runs the tests for the features they work on. Same either way.

The great thing about feature flags for testing is that you have greater control over who is exposed to your new feature. With a feature toggle, you can expose some teams in your company outside of your own, without exposing the whole company. You can expose 10% of your customers and do testing without inflicting bugs you didn't forsee on 100% of your customers. If there's an emergency, you can limit the damage radius with a well designed feature flag. Branches can do this only to the extent that the merge structure allows it, and they can't help with customer testing.

> It is so far my suspicion that most people who speak of feature flags as an alternative to branches either work with trivial products [...] or talk out of some theoretical conviction which has yet to meet with reality

I think I know why you feel that way, but that sounds (to me) like you haven't been exposed to (and so can't imagine) workflows outside of git branching. I mean no insult, but you are jumping to conclusions from lack of experience. Feature toggles as a pseudo-branching workflow predate git by decades, they are used on the largest codebases in the world -- did you miss the comments in this thread from MS and Google?

Feature flags are used in nearly all repos where branching exists. It's not one or the other, you can use both workflows at the same time. A/B testing is based on feature toggles, and all sizable websites on the web are doing A/B testing constantly. Chances are high that you use them already and just haven't yet recognized that they're the same thing we're talking about. If you have config files for your project, then you have feature toggles. If you ever change a value in one of them, then you could instead branch your code, remove the config variable and rewrite the code that used to reference the feature flag to be hard coded instead. Is that making sense?

FWIW, just to put some specifics and back up my own 'conviction' with real world examples, I've been on teams that used feature toggles to test and ship features during the production of several animated films, movies you probably saw. When I worked in film, it predated git. We had a hierarchical repo, but there were no branches, only pushing to master and feature flags.

When I worked in games, the team I lead used feature flags to support cross-team testing on a $1B console franchise.

At a certain web company, in addition to constant A/B testing, my team shipped features to customers who signed up to be on the 'beta' versions of features before they were rolled out to all of the >1M users. We used feature flags to control which teams internally could see & test certain features, and we used feature flags for either long-running or large cross-team features when branching & merging would cause undue amount of branch noise.

And at my own startup now, I prefer to wrap my features in feature flags even when I'm branching. That way I can back out a feature when it causes problems or has bugs without having to do git surgery or even risk a merge conflict. With a few thousand active customers, this has saved my ass multiple times.

[xorcist]

> If it's entangled, the branching analogy is a merge conflict.

I like this observation. As long as two developers touch the same code, you will have a conflict. Now your choice will be where to best place that conflict. In an SCM, in conditional statements, or somewhere else.

> For testing, the prod tests run the prod config, which is likely with all optional feature toggles turned off.

I don't understand this. If you are going to activate a new feature in prod, surely you test it first? Or do you start testing all over again before you switch on a new codepath? Feature toggles must also include externalities such as backend systems changing their APIs. Even if you are forwards compatible, you must take care so your application doesn't break when that happens, and that means testing in advance.

If you really follow the master-is-production branch free model all merging must cease before you activate a feature in prod? That doesn't sound very practical.

> Do you run all possible branches at the same time? Is it an exponential combinatorics problem to have multiple branches? Not really, each team runs the tests for the features they work on.

A team can not be responsible for testing only their own development work, they must also know that it works with other people's changes that might activate before theirs does. That cross team communication is hard enough using branches, but at least teams do not disturb each other's testing. That enables asynchronous development. In a master must always be production model, which seems to be what most branchless development people seems to advocate, every combination is blocking. That's the difference.

> The great thing about feature flags for testing is that you have greater control over who is exposed to your new feature.

Right. I'm not saying that feature flags are useless. I'm just questioning that it can be used instead of branching in non-trivial projects. There are a lot of use cases where it makes sense, for example user interface stuff where branching unnecessary.

As you point out, feature flags are just a new term for application configuration. We've had that forever. It is as an alternative to developing in branches I question it. We generally don't celebrate making everything an option.

> I mean no insult, but you are jumping to conclusions from lack of experience. Feature toggles as a pseudo-branching workflow predate git by decades, they are used on the largest codebases in the world -- did you miss the comments in this thread from MS and Google?

Oh, absolutely. I type this because I want to hear from people with experience. What I meant by "my suspicion so far" is that the experience with branchless development I've heard so far is either trivial or theoretical. There's a lot of pointing at Facebook as a good example but not much in way of first hand knowledge.

And yes, I did notice the comments. The Microsoftie called it "if-statement hell", the Google commenter said it "impacted developer velocity" primarily because of the testing requirements. The former can perhaps be explained with culture, but the latter matches my experience exactly. These comments concerned feature flags in general however, not branchless development which is a special case of it.

Your examples from the game and film industry sounds like it could be made workable, but I imagine that development looks different from the average ERP system or backend service. Certainly with smaller startups you shouldn't overcomplicate matters.

[solatic]

> then I delete the if statements once the feature becomes permanent and gets released to all users

Doesn't this throw out some of the point of feature flags? That if your production servers are getting hammered, and you're having trouble scaling (for whatever reason), that you can degrade service by toggling off various features which are more resource-intensive so that your service doesn't crash entirely.

Feature toggles implemented through conditional logic seems like an anti-pattern to me. Developers make mistakes, and it's very easy to forget to wrap work around the necessary toggles and accidentally expose not-ready code in production, which may also have security consequences.

It seems to me like a better pattern would be to feature-toggle on an API level, and force developers to plan and commit to relatively stable APIs when they deploy new features to production. This is achieved through an API gateway model - a versioned API is introduced to the gateway, and new versions of services must pass integration testing proving that they adhere to that API before the API gateway will serve the new version of the service with traffic. Developers may introduce new code, and new unstable APIs, and deploy them to production, but they will not be user-facing until a new, stable API is pushed to the API gateway to serve to users. This preserves developer flexibility in pushing new work to production, without exposing that work to the user, until the feature is stabilized and ready.

[dahart]

> Doesn't this throw out some of the point of feature flags?

Not if you're using feature flags as an alternative to branching, nor if your feature flag is intentionally temporary. A branch is temporary, and merging the branch doesn't eliminate the point of branches, right?

For performance toggles you want to keep and use, then definitely just keep those.

> Feature toggles implemented through conditional logic seems like an anti-pattern to me. Developers make mistakes, and it's very easy to forget to wrap work around the necessary toggles

This is a valid concern generally, but in my experience doesn't detract much from the reasons to use feature toggles. In particular, the good examples you brought up (performance features & API feature toggles) you can't have without conditional logic, so there isn't any choice or alternative, right?

> It seems to me like a better pattern would be to feature-toggle on an API level

That's a good idea when the feature in question is an API feature. Are you a backend kind of person? ;) Lots of features are frontend, lots of features are full-stack. It all just depends on which feature, but generally speaking having API level feature toggles in your toolbelt is a great idea.

[twunde]

Most languages have FOSS feature flag libraries. One that comes to mind is waffle for python/django.

[hoorayimhelping]

The expectation of a bunch of weird branches making everything confusing (I had that too) ends up being unfounded. You typically only see a few places with branching logic, and people usually draw attention to it being branching logic for a feature or experiment.

How and where you branch really depends on the feature. If you're A/B testing a button color, you'll deploy a different strategy than if you're trying to introduce new logic with minimal regressions. But you can plan out your strategy in both cases to minimize leaking information and disrupting user experience (and you should).

>Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this?

The general flow is: get status of features from source of truth. Check current feature flag's status. Do something based on that status. You can make it as complicated or simple as you like. You can have a global boolean flag for all features where it's always either true or false. You can have different flags for different users, you can have certain percentages of users have different values for flags. But it all starts from getting the value of a flag for this current request from the source of truth.

Let's say you're writing a crud app on an express server that communicates via http to a database service. Your database service is rolling out v5 of their api, and they're at the last stages of their beta so they're pretty confident. You decide you want to test it by routing half of all the traffic to your crud app through v5 and the other half as normal through v4. You could lay out your code in several ways, and you'll have to figure out what works best for your model and your team.

Let's assume you have a feature service you query on every request (maybe it's express middleware) to see if an experiment is enabled for the current user, and you've queried your current user against api v5, and you have a boolean variable indicating whether to show v5 or v4 and it's used in a route handler for your home route.

You could change the url you hit based on the state of the experiment, provided the interface to the api is the same.

    const routeUrl = '/api/v4/home/'
    if (featureIsEnabled) {
      routeUrl = '/api/v5/home/'
    }

Inside the route home route handler you could have an if/case statement that did a completely different request and response.

    if (featureIsEnabled) {
      return queryV5('home').then(respondToV5).catch(handleV5);
    } else {
      return queryV4('home').then(respondToV4).catch(handleV4);
    }

You can also do this on the front end - a lot of feature / experiment services have front-end libraries. I prefer to do it server side.

[aaimnr]

> Let's say you're writing a crud app on an express server that communicates via http to a database service.

As a side note - a 'database service' that's only job is to serve as an HTTP wrapper over DB seems like an anti-pattern. Service boundaries here are not related to any business concern (if you're thinking along the SOA lines) and rewriting already existing DB remote interfaces via http seems like overengineering (if you're doing n-tier architecture). Some additional reasoning on the matter by Udi Dahan: https://vimeo.com/113515335

Would be interesting to hear your thoughts if you think otherwise.

[tigershark]

Using properly interfaces and implementations with an IoC container solves this problem without causing any messy if-else leakage throughout the code.

[latencyloser]

The project I work on at Microsoft uses feature flags. I see it with many Azure services as well. I can testify that it is absolutely if-statement hell, at least in the areas that rely on these feature flags. However, I will say I think 'lazy programming' is more the issue than the flags themselves. For example, isolating feature specific behavior to it's own impl and using an abstract class to share common behavior--that sort of basic stuff gets skipped in favor of 'time'.

Thankfully these are all obscured from the user for the most part.

We have a 'control panel' like thing that we used to enable/disable feature switches. It works well for its use case and let's us enable/disable with as much granularity as we want.

[kostarelo]

Can you give us some examples of those feature flags?

[latencyloser]

Hmm. I can't get into detail. But we use it for things that are mostly 'experimental' for the customer. Let's take the all-to-common TODO MVC app and say I add a caching layer between the app and the database. I would likely add a feature flag to enable or disable retrieving or storing results in the new caching layer. Or, I add a new button that lets the customer multi-select TODO tasks, that would have a feature switch, etc. The reason for this is that it lets us enable these features for customers who either insist on having them or are excited to try new features without exposing the entire customer base to the risk of these features.

The big problem with this is: you end up having 'if(feature)' type statements _everywhere_ for each of those experimental buttons or extra components or whatever. It creates considerable debt that the team has to be accountable for cleaning up as those features start to stabilize.

[dahart]

> Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there.

This might be just a terminology problem. Any company that does A/B testing is using feature flags, so pretty much most of the web is using them. I'd bet more or less all programmers have them in their repos too, maybe you just call them something else, or didn't think they could be called feature flags. If you have a config file, chances are high that you have feature flags already.

For tooling, I get the feeling you're thinking of feature flags as something other than your config files and code constants. But that's it: feature flags are really simple. Add a boolean const defaulted to false in your config somewhere, and wrap the code in an if statement. When you need to A/B test or release incrementally, that's when you need extra non-trivial code around features, and that's often not something someone else can provide. Turning the feature on and off requires specific knowledge of your project's stack and UI.

[kccqzy]

I think there are many different equally valid ways to use feature flags. We use feature flags to slowly ramp up the release of new features only. We have per-user feature flag settings too.

We have a small team so the number of new features that need to be gradually released is very small, and won’t make the codebase a mess of `if` statements.

We don’t really worry about feature flags leaking to users because when we start to use the feature flag, there is already a group of users being able to use the feature (usually employees because of dogfooding, and then gradually to a selected group of power users/VIPs). The frontend JS code is already minified beyond recognition (we use Google Closure compiler in advanced mode) so that’s not really a concern.

Since feature flags are per-user, you can think of them as a column of the users table in the database.

[kcorbitt]

At Google every significant feature in our codebase was behind a feature flag. Pro is that you can roll out features slowly and do a statistical analysis of their impact on user behavior in a well-controlled way. Also, you can turn off a broken feature really fast without having to wait for a new deploy (especially important with native mobile apps).

The big downside was that it significantly impacted developer velocity. You have to test that everything still works with/without your feature enabled. Also, sometimes supporting the old and new way of doing things simultaneously requires a nasty hack, and when you go back in to pull out the old code after a successful deploy you don't actually bother to rearchitect things the way you ought to.

For Google they were probably inevitable, but for a small to mid-sized company I would avoid them.

[QuercusMax]

There's used a ton at Google, except they're called "experiments". I believe that term comes from A/B testing, which they're still heavily used for, but they're also used for feature rollouts.

[mabbo]

> How do you prevent your codebase from becoming if-statement spaghetti?

A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch, and launch fully or remove the feature; consider 'hiding' the ifs behind factories that build the objects that implement the different logic; also, if you have 20 features is development for the same area of your codebase, worry- you may be trying too much!

> How do you prevent new features from being leaked to the user?

I haven't had to worry about this very often just based on my projects, but there are strategies. You could be sending the user a different version of the js/html based on the feature flags (preprocessor as you said). Haven't had to do that, but woah that would be a fun little project.

> What do you use to control feature flags?

My favorite implementation was an S3 json file that effectively encoded a decision tree based on variables used. In code, we could say "here are the five variables about this request, feature object are you enabled?". By modifying that json file, you could change the features state at run time. (Note: this was not perfectly implemented/designed by me and caused a few issues when we first used it. Oops.

But you can do very quick solutions too, read a file or make an object that decides based on non-dynamic logic.

[jasode]

>A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch

I think a big factor that's unspoken about "trunk-based-master" development is that it fits better with "website apps" such as Facebook and Etsy. The "live website" can be thought of as a "single executable" and trunk-based mental model maps well to that. The feature-flags as a replacement for branches doesn't result in an unusable combinatorial explosion. There are minimal # of "alternate universes" of Facebook... maybe a "Facebook for internal engineers" etc. You can tame that with feature-flags.

However, for corporate development where the output is "exe" files... e.g. version 2.x of product has dependencies on a old version of a library and a version 3.x uses a totally different architecture with different conflicting dependencies... you can't model that code evolution in a clean manner with feature-flag "if" statements. If you move the "if" statements outside of the code and into preprocessors (#ifdef) or the "build" system such as cmake, you've just shifted the problem around instead of solving it. Branches instead of feature-flags are cleaner and more sane for certain domains.

[mabbo]

Fortunately, exe development is becoming a small subset of software these days.

[hyperpallium]

Dependency injection? The conditionals are then virtual (which implementation of an interface); the same controls apply to front-end code; global dependency configuration.Though all that infrastructure and boilerplate for explicit modularity is probably only worthwhile at Facebook-scale.

With each feature in a separate file, there is no merging of features, and if you squint your eyes you can see each possible dependency configuration as a git branch, selecting certain files out of a repository, just at runtime. There are no file-level merges, only different files, and different configuration selections. (Of course, modifying code to use the interface in the first place is a file-level change.)

BTW I've been thinking about this problem in the context of computational fluid dynamics, where you want to try different scales, fluid and container initial condition profiles and boundary conditions, as well as different discretizations, schemes and variations thereof. Basically, experimental cases (a bit like "functional test" setups).

[krzyk]

I've worked at Sabre some time ago and we used feature flags intensively.

Feature was enabled/disabled using our own configuration server (which was used to configure many other things) - it was basically a dynamic file with booleans, that we could turn on/off. After the feature was released we removed the if statement from the code.

[NicoJuicy]

Who says no one is embracing them? There are tons of tools available if you check for it. But you should google for "feature toggle" libraries and not feature flags

[guelo]

I've used feature flags at 3 non-facebook companies.

[baylisscg]

Feature flags _are_ widely used. They generally manifest as commented out code. Like many features that migrate down from these massive codebases it's a common behaviour thats been systematised.