| I am technical lead for one a collection of risk systems and trade storages for one of largest banks in the world. We have simple rules to prevent significant data loss: 1. Deleting data is not permitted and mutable objects are highly discouraged. No deletes/modifications == greatly reduced possibility of data loss from app code errors. I lied a little bit. There are functions that can remove data, but they are hardwired to refuse to work unless it is beyond question that they can't remove live prod data. For example, the function will refuse if the collection is not prefixed with "tmp" or "test. For production objects we have a system where we basically do CoW and create new versions of the documents and vacuuming system that archives old versions. The archive is preserved for a minimum period of time (2 weeks) to give chance to react in case somebody makes a blunder and screws up some rules to remove too much. 2. Remove write access to the database from every single person. No single employee should have write access to the database. No single employee should bear responsibility of having to work with an account that can let them destroy the database. No employee has access to PROD credentials, which are generated automatically and not present in configuration or application server. Instead, if you have a need to introduces some changes to the database, an app was created where you can write your code as a kind of job that can modify the database. The job is not allowed to take any parameters (only has a name) so that it is possible to audit what it is going to do exactly. This code then goes through regular development process including code reviews, automated tests, etc. Once deployed to PROD you can go to API and execute the job by name within PROD context. Data loss prevention is a significant issue that no CTO should "dump" on his/her employees. Also not accepting responsibility for everything that happens in their department is a sign of lack of leadership. |