|
We've been discussing potential solutions, and came up with three avenues, listing their pros and cons. We're leaning towards a "Sandbox accounts in production" approach at the moment, but recognize that none of the solutions are free, and whatever we decide on will have a big impact going forward. Production-like environment + Clear, separate environment
+ No impact on production data / users (events, emails)
+ Not a problem if we mess up data without cleaning up afterwards
+ Runs the same code as on production
+ Requires very little changes to existing code
- non-production code can (and will be!) deployed, making it no longer "production-like"
- data needs to be kept in sync, to be usable by clients
- every service needs to run both a production instance, and a production-like instance
- all external services need to work with this production-like setup (payments, emails, etc...)
- higher ongoing maintenance overhead
- requires "manual agreements" on how to manage/manipulate this environment
Individual temporary environments + all the pros of the "Production-like environment"
+ even more isolated, higher guarantee of your expected state of the environment
- requires a lot of CI/operational changes
- requires ongoing maintenance to keep working
- requires "operational" knowledge to make new changes work with this setup
- requires more syncing of data
- higher costs of running
Sandbox accounts with special "capability" flags, in production + All test-data is scoped to a (sandbox) user account, single "source of truth" on wether some piece of data is test-data
+ One single environment to maintain, no divergence
+ Data is always the same as production
+ Whatever code runs on production, is what you test
+ Because you want to test your changes, you automatically make sure your new code works with sandbox accounts / capability flags
+ You deploy your pre-production changes behind a capability flag, for testing
+ A preference page allows you to enable/disable certain capabilities to enable real/fake PSP environments, pre-release functionality, etc...
- Much more complex to realise
- Only works (well) for user-scoped test-data
- Might result in higher learning curve to ship a feature that works with sandbox+capability flags
- Production data can be changed accidentally, there's nothing but our own code between test and production data
- Other systems need to be able to handle (and/or ignore) test data
- Only suitable to the specific use-case of testing user-flows, not for testing f.e. if a library upgrade broke anything
|