This post comes at a great time! I've been wondering what folks prefer - git or DB backed configs and feature flags. How do you decide to go with one or the other?
What do you guys do when GitHub inevitably has an outage?
Nice thing about git feature flags: the state of your system is all in one place. It changes only when you do a code release, which includes feature flag values, or an infrastructure release. You can easily see, just off git history, when the state of the system changed (and what changed), which makes incident debugging much easier.
With DB feature flags, there's one more source of truth for changes to production infra.
(downside, of course, is that changing feature flag values is much slower using git vs DB)
This! 100% All the same benefits of config as code, infra as code. And with feature flags if something goes wrong its a simple `git revert` to get back into the previous state.
Another benefit is you can easily replicate the current (or previous) state of production/staging/etc flags locally just by doing a `git clone` and then run our self-hosted version locally. Its a single binary, can be installed with curl or homebrew and can read the flag state from your local filesytem.
This allows you to test your code locally or in CI with the same state in production
Maybe I’m not fully understanding what this product is doing. For something like configs/feature flags, it imo should be dynamic.
I was thinking that git (in a separate repo from main code) would be used to store the changes to configs, but then you would still need a system that tails the git changes and distributes them to clients.
That’s the way config serving was done at Facebook - a mercurial repo with all configs and tooling to edit configs which creates mercurial commits. Then, the mercurial repo is continuously tailed and values are saved to ZooKeeper, and then client libraries read config data from zookeeper / subscribe to updates / etc.
Flipt is live tailing the repository and serving this dynamically to the clients.
The repo with flag configuration can be solely for flags, or alongside other infra configuration on in more of a monorepo. You decide how you want it setup.
Obviously if it is alongside code, you may to contend with CI in order to validate a change. But with the rules in CI or other monorepo tooling, what runs and when can adjust this behavior to improve time for configuration to become live.
Once a configuration change is integrated into a target branch in the repo, then it becomes readable for Flipt and servable once fetched.
I worked with a mesos config app, backed by git. Watching the whole go down when github died resolved none of us to ever trust it again for that sort of role.
On this, we support publishing the state to object storage (S3, Azure, GCS) and to OCI as well in Flipt.
Flipt Open-Source can be run to consume from these locations. You can go as far as configuring a workflow to publish on push, so that you can combine our managed UI with any of these distributions methods through Git.
With any of these backends (including Git), we periodically fetch and cache data in-memory. Evaluations work on an in-mem snapshot. So temporary downtime doesn't propagate into your applications being unable to get flag evaluations.
Caching in this scenario isn't something I'd lean on unless you can invalidate or repopulate easily. I've used etcd in a feature flag scenario due to the speed of it's replication and it's ability to queried frequently without the need for caching.
I tried to answer your first question below, hope I did!
Re: GitHub outage, each org gets their own instance in our cluster and maintains a checkout of the git state, so you can still write/read from your environments, they just wont be synched to GitHub until they recover.
We're also thinking about adding other 'sinks' like S3/object stores and OCI as backup sinks.
With DB feature flags, there's one more source of truth for changes to production infra.
(downside, of course, is that changing feature flag values is much slower using git vs DB)