Hacker News new | ask | show | jobs
by lucastech 1651 days ago
This feels very timely, as I just encountered it yesterday.

The steps I took were:

1. backup production before making any changes

2. import the production backup locally

3. test the script locally, QA the result to ensure desired outcome

4. repeat 2-3 until you feel confident that you're able to consistently get what you need from the script

5. wait until a low volume time for your platform (if possible), run a backup, import and test 1 more time locally, apply the change in production

However, after doing this, a coworker noticed something we had missed when QAing the data. As a result I had to revert the changes, I had a rollback query ready to restore the modified records.

As part of the planning for this, I added a new column to the table in question to track if/when a fix was made to the record. This allowed me to identify the records which were modified to roll them back. While this isn't always practical, it made sense long term for our situation.

A secondary option I've used in the past is exporting the table to be updated and then writing a simple script with the modifications necessary to generate the SQL to update each record individually. I've then used the same script and export to build a revert update query as well just in case something unexpected comes up.

At the end of the day, make sure you're backing up before any changes, plan ahead in case of unexpected errors so that you can revert quickly to avoid downtime instead of scrambling when it goes wrong. Test it a few times locally (change + revert) to try and sort out the issues prior to applying them in production.

edit: formatting