Ask HN: Handling customers that want feature previewing?

[cbono1]

Curious how you folks manage the release process with larger customers that ask for advance notice (ex: 2 weeks) or previewing of any features before release or general availability?

We have a single sandbox environment currently, but different customers are asking for varying amounts of delay before feature release. How do you manage bug fix deploys in these cases as well?

[Arubis]

Most of the discussion here will be technical, especially around feature flags and friends. That’s fine, but please don’t miss out that this is a clear signal to charge these customers a lot more money.

[talolard]

I'll offer an illustration. We sell our SaaS at X. We have a customer that needed stronger SLAs and servers located in Europe. They pay 8X and (I think) both sides feel like they got a great deal.

So to paraphrase Arubis, this is a business question,not a tech question. At what price point is the technical investment covered ?

[ambivalents]

> a clear signal to charge these customers a lot more money.

Interesting, can you explain this a little more? Some of our customers chomp at the bit for particular features, and when we can we give them early access. We do not charge them more for arguably huge amounts of value add to the product.

I wonder how to have these types of conversation, internally and with users. I'm a PM but have a direct line to management. I'm frustrated with how much money I believe we are leaving on the table, even if it doesn't directly benefit me.

[avalys]

I don’t think this is about charging for new features - it’s almost about the opposite.

I read this as “A customer that asks for advance notice of feature changes is likely to be a big enterprise customer that values stability and predictability, and the fact that they’re asking suggests your product is important enough to their process to matter, so you should take this opportunity to charge them a ton of money for an ‘enterprise support plan’ which includes advance notice of new releases, etc.”

As the GP said, it’s a signal that they’re a certain type of customer.

[feifan]

If you also have a direct line to your users, you could try asking them how much they'd be willing to pay for either some new feature that's about to come out, or for the right to early-access. Get maybe 3–5 data points and you should have a pretty solid case to management (either to charge retroactively for existing features, to charge new users for existing features, or to charge all users for upcoming features).

[wonder_er]

rather than ask "how much they'd be willing to pay", it might be more compelling to simply say "It will cost $X for early access, limited to X slots, in exchange, {we do such-and-such}."

If they say "OK, great", you've got immediate revenue to bring back to your management, and a strong case to continue doing more of the same.

Customers routinely over-and-under-estimate future value, compared to what they're then willing to pay.

They're often wrong in both directions, they usually won't pay as much as they think they will, _and_ they'll often pay far more than they think you will.

There are many, many reasons for both kinds of errors, many of which can be de-risked in advance and safely ameliorated.

[exolymph]

As a smaller-scale contractor I just tell people what I charge. If they're not willing to play ball* get to premium compensation, no deal. It's cliché but BATNA is everything.

*I'm willing to play ball too, e.g. for a staff position I might take extra vacation in lieu of more cash, tradeoffs like that. But basically I'm not gonna take on extra hassle without being incentivized — because I don't have to and don't want to, all there is to it

[rm_-rf_slash]

If a customer requests a changelog in advance that’s one thing (should be as simply as fine-copy-paste-email), but setting up a preview sandbox is work that should be negotiated by a contract if it wasn’t already signed or in the terms of service.

[zerkten]

You are right that previews can be a time sink. It justifies some strategic thinking because the blast radius can be big.

Someone can forget to have a separate set of terms and suddenly the support team are on the hook for supporting previews to a greater degree than expected. If features aren't locked then suddenly features get jammed in because the preview is a good hook for customers who haven't yet acquired your product.

[pjc50]

I think the ways to approach it are:

- this incurs real costs, so we should charge for this. (The charge doesn't have to be related to the costs! But if you're doing something extra there should be an extra line on the invoice)

- this customer wants something that other customers don't have. There's all sorts of real-world services where you can pay to board the plane first or get to the front of the concert. These may be several times more expensive than the regular service. That lets you phrase it in fawning terms ("you can join our Super Elite Ultra Platinum Preview Fellows") and thereby extract much more money.

[lmeyerov]

enterprise support contracts provide access to early release builds / feature flags / etc, or require signing up to be a design partner (ex: logo + joint talks)

[williamdclt]

That's customer support effort: you have to have a separate feature release communication for these beta users, and very probably some support when things go wrong because they're using a beta product.

That's also engineering effort: you need to maintain feature flags or a different environment, which is not cheap in the long run

[Macha]

Feature flags is usually how we handle this. We have the flags anyway for developer use, so making them user specific was not that much incremental effort though we have had a few cases where old workflows were not removed when they should have been.

We did once have a customer with their own environment with a different version deployed, as they were initially a major client with such demands but after a few years it became clear it wasn't worth the maintenence effort to special case this one client so we made it clear in contract renewal one year that this would be going away.

[notJim]

+1 on feature flags. Set up your flags so you can disable features for certain customers, or groups/classes of customers. Delivery and release can then be separated. Release frequently, deliver features when it makes sense.

Also recommend making sure customers know that this is time-limited, otherwise you end up supporting too many different configurations of your product.

Also if these are customers you're in regular contact with, you can preview your roadmap or demo features that are coming soon. This helps them feel in the loop, and gives them a chance to provide feedback.

[williamdclt]

That's about the only answer, although feature flags are hell. Once you have a feature flag, you now have 2 versions of your product. 2 feature flags, that's 4 versions to maintain, etc. Make sure you don't conflate "feature flag" and "setting", and make sure you clean up these flags when they're uneeded (and actively push the business to make them unneeded!).

Fowler has a good write-up about different types of feature flags: https://martinfowler.com/articles/feature-toggles.html

[petters]

Agree on this. We delivered new versions to customers once a week, but every new version should not have been noticed by a customer at all.

The roll out of feature flags is a separate process.

[tombot]

+1 for flags, you can also put features into production earlier, safe knowing they will only be visible for the users / teams you’ve added to the list

[1a527dd5]

+1 on feature flags. I find them to be a pain, but fussy customers insist "that we need to test it in staging first".

[geocrasher]

Sounds to me like the customer is trying to solve a problem and is coming to you with their chosen solution. My suggestion: Find out what problem they are trying to solve, and address that.

Also, you might look at cPanel's notices that they send out where they explain upcoming feature updates well in advance of the rollout. They also have different release cycles, where the newest features are released to the EDGE cycle instead of the STABLE, and then the long term release cycle remains unchanged except or security updates.

[mooreds]

> Sounds to me like the customer is trying to solve a problem and is coming to you with their chosen solution. My suggestion: Find out what problem they are trying to solve, and address that.

This.

Problems they might be trying to solve:

   * want to be able to jump on new features as soon as they are released, but need to plan with the eng team
   * want to be "in the know" so not surprised by new features
   * want to be able to give feedback and help direct the roadmap
   * have been bitten by bugs in your software (or other vendors) and want burn in time, especially if you upgrade them without their choosing (as is typical in some SaaS models)

I'm sure there are others, but it's worth asking why they are requesting this if you haven't done so already.

[zippergz]

Honestly some customers I've encountered have asked for stuff like this (and similarly meetings to go over product roadmap) almost as a power play. Either to see if they can get us to do it, or to feel like they have control over our development. If you're small enough and the customer is big enough, you might not have any choice, but I resist it pretty strongly unless it would be financially ruinous to lose the customer (and we have never lost a customer over refusing to do it).

[stronglikedan]

Reminds me of when Disney wanted us to do a considerable amount of unpaid work, simply because they were Disney. We don't do business with Disney any more.

[void_mint]

In retrospect, was it the right move?

[plasma]

Have you clarified why they want this notice?

In my experience it’s because if you roll out a new feature (or non-trivial UX change), that can wreck someone’s day and cause internal support issues into the company that uses your product.

Consider your rollout of new features to be least impacting, don’t confuse someone who’s using your product to get work done by changing something that breaks a workflow that worked yesterday etc.

You can put new features in new sections, or need special users to activate them.

In some cases we would show screenshots of upcoming features to hear about any other feedback, but otherwise focused on new features not being disruptive.

[cbono1]

Thanks for the reply - yea these are great points to consider and raised a few of them. Ideally rollouts benefit all customers, and the concern is that there is some possibility that something might not jive with them and they want the ability to do testing ahead of time.

Seems like putting as many things that might risk this under settings/configurable as possible? This does extend development to have to gate everything in this way though.

[plasma]

In my experience it was just not doing huge UX reorganisations without prior notice.

I’d dig deeper into the why with your customers, after we understood moving buttons around causes people’s day to be hard we did it only when really justified etc.

I wouldn’t hold up releases for customers, but do dig into the why more with your customers about why they want this notice, you may find it’s just an assurance something like a workflow isn’t totally different next Monday morning.

[dsr_]

Context: financial SaaS of interest to banks, brokerages and RIAs; no direct-to-consumer at this time.

We have two special case preview environments. They're set up just like production, but with lower capacity and no SLA.

The first one is general previews of what we're planning on releasing; it's a normal part of our release process and every release is there for 2-14 days before going to production. All customers have access; some use it extensively, and some basically ignore it.

The second one is for specific in-development features; in theory we can spin up multiples of this, but in practice we don't want to diverge too wildly. Generally we showcase one feature at a time, and we gate the features by customers.

[theoblank]

I would seriously consider not allowing special treatment in cases like this. In my experience things like this can be a slippery slope. Decide what is most broadly reasonable for both your company and your customers, communicate it well, and then spend time and energy on your core product.

[Jugurtha]

Figure out why they're asking that. This appears to be an "XY problem"[0].

An example of this is a friend asking me how to weld a wire to a thin metallic plate. I digged further to find the actual problem for which his solution was to weld the two. He said it was a fuse's wire that melted and he wanted to weld a bigger wire to avoid that, to which I replied: don't. The whole point of the fuse is sacrifice to protect down stream equipment.

The same when a relative asks how to format a disk as their solution to solve a problem.

Generally speaking, implementing customers' solutions is a recipe for disaster, especially with enterprise.

Ask what it is they are really trying to accomplish, and figure out a way to solve that. Listening for problems instead of solutions or implementations.

- [0]: https://en.wikipedia.org/wiki/XY_problem

[browningstreet]

I had a similar situation in fintech. We had 3 different release cycles and hosting farms for fast, medium and slow accepting customers. Fast went to the customers who didn't require approvals, and who were flex when it came to errors in the release. Then there was a medium channel where the other customers were fine as long as a couple of the major customers in that pack approved things. The slow channel basically was for customers who had large validation or training requirements associated with releases.

[cbono1]

Would this break down if you had 10 customers that would prefer the slow channel of release, but each had their own ever so slightly different desired release pace or training requirements?

[midnightmonster]

Key word "desired". Make the slow channel the longest reasonable time that any of them want. If they want shorter, well, there's medium channel or slow. You pick. If they want even longer, well, there's slow channel, and it's good enough for these 10 enterprise customers.

[browningstreet]

Yes.

There are always requests and negotiations in play. But those three channels sufficed almost all the time. At times we'd combine channels if there was an urgent need and agreement from customers.

The slowest customers were the most enterprise-y. Things almost couldn't go slow enough for them. When there were complications to the release process, they could be a release version or two behind and there'd be a catch-up release.

We also had customers outside all of this, and had custom releases, and we charged very high fees for this. Ultimately, we were able to bring them back into the slow release because the extra work wasn't really worth it. Part of the requirement for these kind of customers is to have additional engagement efforts to make sure their internal teams were really aware of the details and contingencies of releases. Big companies move slow and need over-communication, and you can't trust them to do it themselves. It was more effective to take that responsibility on ourselves.

I preferred to take the things that were difficult and do them more often. Keeping a cadence to things kept everyone together. The minute you break cadence -- and entertain special circumstances -- it's really hard to get back on cadence, and that cadence includes testing, notifications, documentation, feedback, approvals, releases.

[cweagans]

"Thank you for your inquiry. This is not a service that we offer at this time -- our testing and release process is fully automated and there is not a way for us to exclude some customers and not others.

Can you please help us understand the purpose of the advance notice/release delay is? What benefit does a process like that provide to you or your organization?"

---

I generally use some variation of that ^^. Having some customers on different release tracks is very painful in my experience, especially if it's done without properly planning for it in your release process. For example, let's say that right now, anything committed to `main` goes out to production via some automated process. What happens when you have a security bug that needs dealt with ASAP? In my org, this usually means that the security patch gets rolled out to everyone, and then when it's time for customers on a slower release schedule to "catch up", there are conflicts to resolve manually.

It's just not worth it. Figure out why they're asking, but say no. If it's a problem with buggy releases, that's the _actual_ problem to put some time into solving.

[johncessna]

The argument I always got while trying to argue against delaying features for large customers was that it causes a training headache. If you have large enough customers, redesigning a UI triggers a training initiative for the 100s, if not 1000s of customers that use your software.

[cweagans]

That's a fair concern. There are other ways to address it though. For instance, release the new UI as a "preview" that users can switch to if they want to. If they hate it or can't understand it, they can switch back. Even better: redesign the UI in smaller parts and roll it out slowly over time.

[ghiculescu]

Lots of good answers below on how to make it work, how you should charge more, etc.

Remember that saying no is also an option. You aren’t obliged to say yes just because someone asks for something you don’t sell, even if they are willing to pay.

For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”. I’m not keen to guarantee that, so it’s a simple no, sorry, we can’t do that. It’s yet to be a deal breaker.

[cosmie]

> For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”.

If you frame it as early access, then people are apt to try to throw their wishlist at you, thinking it's still "not official yet" and easy to change.

But based on what OP asked, I'm not sure that's the primary reason. It seems more like his larger clients are finding the changes disruptive enough that they're wanting advanced notice to prepare for them. So integrating a lot of the suggestions here can very easily address that issue, while also providing leverage points for client management and sales.

@OP - don't think about it in terms of preview access or letting them into your internal sandbox environment(s). But rather, restructure your releases to decouple the release from the account migration. Using feature flags and such then allow end users to determine whether they immediately migrate to the new feature when it's available, or if they continue with the status quo and wait a few weeks to familiarize themselves with the changes before opting into them.

For end users – this gives them room to both continue with their day-to-day needs using the process they're already familiar with, while giving them flexibility to familiarize themselves with the changes and prepare (in whatever form they need) before committing to them.

For the tech team – there's an initial lift to adjust the app architecture to support this type of usage, but insulates them these sorts of client requests in the future since the app now supports these types of opt-in migrations within the live environment. If a user wants a sandbox account, it can be provisioned as a normal account in the live environment but with sandbox-like restrictions applied to it.

For the sales/account teams – this can be used as a negotiation point. You can position it as normal accounts get auto-migrated to new features upon release, but enterprise clients can pay for the ability to control their own release/migration timing via these opt-in feature gates. And a sandbox functionality (which provisions a new account using the current configuration of their existing account, but with applied sandbox-like limitations) allows them to both prepare for it and test it in a controlled environment before enabling it on their primary/production account. That sandbox capability is independent of the opt-in capability, and a second point of negotiation.

And since your application now supports both of these directly in production and doesn't require bespoke work or custom environments from your tech team to support, the incremental cost is pretty minimum to give it to a particular client. Which is hugely useful for a sales team when closing larger accounts. They can give it away without much hassle if they have to in order to close a deal or provide value-add during renewal, while simultaneously capturing incremental revenue from clients that are willing to pay substantially for these sorts of things.

[cbono1]

Thanks for the additional context here.

I think some of the administrative pain can come from having a 'sandbox' or 'qa' tier beyond dev (which in this case would be customer facing) before production.

If data separation isn't required/essential, seems as though a a separate environment is actually unnecessary and feature gating could be used. But if data separation from other customers is required, then new infrastructure should be introduced via separate CICD if I follow correctly?

[cosmie]

> I think some of the administrative pain can come from having a 'sandbox' or 'qa' tier beyond dev (which in this case would be customer facing) before production.

Pretty much. It's hard to guess at the reason why without knowing the product, but for whatever reason customers are wanting access to new features/releases before using them. Which likely means that they need to do something on their end every time you release (validation, update internal training/process documentation, etc). And it's causing disruption when things change outside of their control. So they're wanting more control over that change timing, to smooth out the impact of those releases as far as their usage goes.

If your release codebase can only support a single "production" version of a feature at once, you end up having to provision entirely new environments to hold a client back. And again, for every client wanting this. Because the timing they want isn't going to be convenient enough to be the same.

So instead, structure things so that you can run both the old and new version of "the thing" simultaneously in production, and give the client self-service ability to activate the new version when they're ready. That way your production environment itself is able to accommodate these requests and client's can opt-in when they're ready/comfortable to.

The biggest challenge to doing this tends to be architecting your application so you can push releases to the backend independently to releases in the frontend. Which can generally be handled by routing things through middleware that provides a stable response for a given feature version. If the interface is using the latest version, the middleware is basically just a stub that does nothing. But if the interface is using a prior version, then the middleware accepts the request signature of the prior version, translates into what's needed to execute on the new codebase, and returns a response that's consistent with the prior version. It's a few years old, but here's[1] an example of how we handled the data-model aspect of this sort of setup at a prior company

So in effect you're actually upgrading all customers to the new release at once, but from the customer perspective everything still looks/feels/act the same as they're used to until they "upgrade/opt-in" to the new release.

It's a pretty major lift to implement this architecture (particularly on an existing codebase), but maintaining it is easy to incorporate into the regular development process. And completely eliminates the operational and customer management headaches you're having now.

With this architecture, it doesn't really matter if you're running isolated deployments or a multi-tenant architecture. You only ever have a single version of production in both cases. The "sandbox" isn't a new environment, but a new production account that copies over the configuration/setup of their existing one while applying sandbox-like restrictions to it. This gives them a temporary space to prepare for the change so they're prepared when they flip the switch on their primary account.

[1] https://github.com/TomFrost/Vers

[bkuehl]

There is a lot of talk of feature flags but that only works when you can realistically delay your feature 2 weeks so that the 'special' set of clients can see it early. What we've seen that has worked is essentially demoing the feature in staging/test to stakeholders at the major/demanding client. They are happy that they aren't taken by surprise and then we provide training materials for major feature/UI changes to their users. Never have had an issue, client is happy and essentially zero technical involvement.

[pkrotich]

We use release branches (git) for such cases - mainly because our deployment is based-off release branches, thus gives us flexibility to move a customer to a pre-release or even have then stay at a particular release forever. Bug fixes can be back ported as needed.

Tried feature flags but it introduced complexity.

I wouldn’t charge more as others are suggesting- bugs you’ll address before general availability is worth the trouble.

[cosmodisk]

I really like how Salesforce do it: they make an announcement that a feature/fix will be available from day x and depending on its importance, they then give 100,200,500 days or whatever to install it or it gets installed automatically on the deadline day. This gives plenty of notice to customers to prepare( updates can break some customisation on client's instance).

[zerkten]

You need to consider the blast radius and how long features may be in preview. Ideally, you would scope the preview to just a subset of users. When you go to a complete customer base you start to get into the situation where the customer has failed to communicate with their users, and have more people raising issues. This isn't bad necessarily, but you have to be prepared to handle it.

If you keep something in preview for a long time then there can be a divergence from the main product. Make sure that data or work from users is migrated across when you finally release.

It makes sense to think about the opportunity for support to participate in the preview. Let them listen in to customer calls and learn about the problems. Providing a dedicated support person for previews is tempting, but long-term it's better to avoid this specialization and enable the whole support team to participate. This shares knowledge and avoids certain people being granted special privileges while others toil on average support tickets.

Microsoft publish some of their methodology at https://docs.microsoft.com/en-us/microsoft-365/admin/manage/.... The key point is that they have set out the boundaries for their program so that customers know what to expect and it can be repeated for every customer.

You will have to learn how to do your previews, but you should lay down some rules. Set it up so kudos from participants can be used for marketing after. You are the one to decide when the preview ends, but setting some minimum time in preview gives customers more confidence and time for you to process feedback. Make it clear if features are locked when you enter the preview and what feedback you are after.

Once the preview is over, thank the participants. Call them out if you have a public forum. Participation can become a mini case study for your marketing efforts.

[leesalminen]

I also had a handful of customers like this. We ended up creating an opt-in “beta” channel that we treated as a first-class citizen along with “prod”.

Basically we had 2 production stacks with CI/CD for each. The only shared thing was the AWS ALB. Our customers each have their own subdomain so we used that to point to different EC2 ASGs.

We would continue pushing bug fixes to “prod”. Once a beta period ended, we merged to master and re-deployed master to both prod and beta channels. Rinse and repeat.

It works out pretty well. Occasionally there’s a really important bug fix that we have to cherry pick from prod to beta, but it’s rare enough to not be a huge concern.

Oh, and, charge them for it. I never got any pushback.

[cbono1]

Thanks for sharing!

Was the beta channel that you had in the pipeline for prod? i.e. prod would have no chance of getting ahead of beta because prod was always deployed based on what was last on the beta release?

Hot or bug fixing is an interesting topic because you could end up with diverging histories if you're not careful. Did you try to maintain `master` as the single source of truth or did you make separate release and deploy branches for each environment?

[leesalminen]

We created rc/xxx branches in git. Developers would merge feature branches destined for beta to the rc. The rc gets deployed to the beta environment. Later, we would merge the rc to master and then deploy master to both prod and beta environments.

Prod would be ahead of beta during hot fixes. Though we try and avoid it, it did happen occasionally.

You are correct that it took some extra effort to keep it all clean and manageable.

[jpmoral]

Do the two stacks each have their own database? Have you had customers wanting to move back and forth between channels?

[codegeek]

Charge them extra for a "staging" environment setup only for them and then deploy all latest stuff there for them to test. Then get their approval to apply it to their production environment. Charge well for this.

[cbono1]

Would you have the staging environment share data with production or have its own infrastructure? or use feature gating? One customer may not care that it shares the same DB, but another might.

How would this work at scale? say if there were say 10 customers wanting approval, you'd have to have separate code release branches/versions for each customer? I imagine this could get error prone.

I suppose to your point on charging additional fees for this you can afford any sort of workflow.

[daniellarusso]

From your description, yes, you would basically have forks of your codebase for each customer and deploy that to a sandboxed staging server.

That would be used for acceptance testing.

Bonus points for having up-to-date data in the staging DB.

I assume this is a multi-tenant implementation.

The other option is to charge the customer to disable unwanted features, in addition to enhanced features.

If it is multi—tenant, having the ability to ‘feature-gate’ could be worth the effort, if you can charge for enabling AND disabling features.

[tboyd47]

You need at least 2 sandbox environments: one that's more development-oriented (little to no restrictions on merging), and one that's more release-oriented (as close as possible to production).

Ideally you have 3, and the third one is not used except to demo new features to customers AFTER they have been released (and perhaps, hidden behind a feature flag).

If you only have one sandbox, then it becomes a brawl between development, sales, and management.

More than three is overdoing it and assuming a technical burden for no benefit. (No, you don't need the capacity to spin up sandboxes willy-nilly. Three is enough.)

[winkelwagen]

Difficult to answer without understanding how you are deploying currently. I do think feature flags can be an anti pattern. I would ask the customers why they want it in the first place. Is it a trust issue? Is it an internal process that they require? Ideally you would want to stay flexible, that would mean not tying your company to their schedule. It is a hard application to deploy multiple times for each customer. Perhaps a more lts approach for those customers might work. Just keep in mind the larger the releases get, the more issues it might cause.

[jitendrac]

I would suggest pre-defined staged rollout based on customer preference. For example, create three category Alpha, Beta, Stable.

Alpha: Gets new features immediately and gets support within 24hours if something breaks.

Beta: Gets new features Every 45 week, and gets priority support if needed.

Stable: Gets new features quarterly, Mostly they are forced to upgrade from old features which you dont want to support, But they will be essential volume of your revenue.

alternatively, you can give them sandboxed demo read-only(resets every night at 12 o'clock) account for testing.

[OriPekelman]

well, self horn here... but that is a basic functionality of https://platform.sh (CPO here). You get per branch full copies of production in isolated staging environments that are very cheap to run. So you just clone production to a specific branch apply whatever changes and grant access to the clone. ..you do need to run production with us for the magic though... and a PaaS is not for all use-cases (specifically when running a SaaS...).

[villasv]

We pick those customers to be beta testers. They receive pre-release features, but in return they must accept possible bugs will happen (of course, when one is identified we give them the choice of disabling the feature immediately).

The schedule is totally in our hands, though. Once we say it’s time to go GA, it goes. At this point there still are feature toggles, though, just in case.

[Hamuko]

If a large customer wants to decide when they get new features, they need to pay for their own dedicated environment that is on a separate release cycle. Otherwise they need to adapt to our release schedules and make sure they get all the previewing that they need on a test environment.

[antonpirker]

We create a separate branch for the customer and preview this branch with wunderpreview and give the customer the link to the preview.

Every once in a while we merge new stuff in the customers branch so she has always the latest stuff in her preview.

[quickthrower2]

We have a powershell script that:

1. Copies the production do 2. Removes sensitive data 3. Deploys the chosen build of that app (from Appveyor) pointing to that db

So in a nutshell CI/CD pipeline

If you are having trouble perhaps there are a lot of manual steps to your deployment.

[runwaykid]

One way is to make use of AppConfig to store customerwise feature roll out plan. To save cost, add 15 minute caching on it (the thick client comes with the local application caching feature)

[mbrodersen]

The solution is to ask those customers to pay for the extra work required. If they refuse then it isn’t important enough for them.

[that_guy_iain]

Seems like you need advanced feature flags.

[cbono1]

In your comment here, what's the distinction of advanced vs. a regular feature flag?

[that_guy_iain]

Regular feature flag is just on and off. When you start getting into the advance you can change it for user groups, specific users, timable so it's only enabled for a timeframe, only on a specific env. The possibilities are endless.

I make the distinction so people don't just think they can put a single if statement in and be able to handle this sort of scenario.

[cbono1]

Got it - that clarifies it.

[Animats]

Is it that the customers want early access to new features, or they don't want you breaking the product?

[bri3d]

Once you know you need to do this (and at every enterprise SaaS company I've seen or worked at, you eventually need to do this - the sibling comments about charging more $$$ are smart though):

* "Release train ahead" - code takes 2 weeks (or 4 weeks) to reach Production from merge, but it goes out to a Preview environment immediately or at the next release. So, you've forked your code into a Prod train, a Preview train, and mainline. You probably need the ability to hotfix to prod and preview fix to the week-ahead-of-prod preview environment, as well as roll normal development forward. You do gain "soak time" for code in the Preview environment, which is a slight advantage. And your QA is easier, because you have three discrete, testable things ("does it work in Prod, does it work in Preview, does it work in trunk"). But, you have another runtime environment which can diverge, which is a big disadvantage.

* Feature flagging by "release train," single execution environment. This is probably the most "Software Theorist Friendly" way to do this. You don't have to maintain/debug multiple environments beyond what you're already doing. Some customers are in the "preview ring" and get more features toggled on. Single runtime environment, single codeline. Fixes are just fixes. QA is a little harder depending on how you set up the possible "rings" and combinations, and the operational blast radius of your Preview code is now the same as your Production code (make a bad query that kills a backend service in Preview code, well, that's also Prod code. Oof.).

* Free-range feature flagging. Some complex arbiter system manages a huge range of feature toggles on a per-tenant/per-customer basis. Having this range of operational knobs is very fun and works great for customers who are particularly risk averse. It also lets you sell SKU-packs of flag combinations, roll specific features to specific "Early Access Programs," and so on. But, you get a combinatorial explosion of feature flag permutations, which depending on how coupled your code is and how intelligent your QA system is, can become a disaster: "Oh no, FooFeature only works with BarFeature on but our QA system always has BarFeature on so we didn't catch the regression! And only one customer has BarFeature disabled for legacy reasons!"

* Some combination of 1, 2, and 3 - for example a Preview environment that runs the same _code_ as Prod, but with the toggles set differently. A few feature flags which can be enabled for early access programs, but a general rule that code needs to live in a release train. This is my favorite approach. Having a separate Preview environment eliminates the blast radius concern of Preview-toggled code breaking Prod. Operational costs are higher, but these can be passed to the customer (making customers buy preview environments as a line-item is pretty popular). But by maintaining single codeline (rather than deploying a fork of last week's code to Prod), the code-theory is still easy, you don't have divergent release trains, and you don't have the combinatorial explosion that comes from a free-range feature flagging system.