I think for many (though not necessarily most) busineses, especially micro-ISVs running a SaaS, the business cost of doing what they "should" be doing far exceeds the actual cost of letting bad-things-happen and paying for the cleanup afterwards.
As the designated "code-and-Red Bull-guy" at the micro-ISV I work at, I'll admit that I've unintentionally nuked the production DB at 2am - but fortunately our cloud database provider could do a point-in-time restore and everything was fully operational again by 2:30am. That's because the cost of setting up infrastructure and procedures to eliminate the need to ever manually run DML/DDL against our prod databases would be... probably a multiple of my salary - and be required indefinitely into the future (as that infrastructure would have to be maintained as the database's design changes over time too) - whereas the cost of having PITR on our prod databases in Azure is... a rounding error.
So yes, our prod is going to go down in future - we can't afford not to, honestly (it's a USA-only B2B SaaS, we get literally zero usage before 6am EST and after 7pm PST).
The college-hire is not the problem; it is the person who gave the kid prod access, or perhaps at a higher level, the person who architected the DB permissions structure. If one person alone can cause a production SEV, many things had to have gone wrong by many other people beforehand.
I remember having a test suite that would connect to a local db running in a docker container and would nuke the tables and then set up the records in a known state before running through. Worked great until someone changed the connection string to point at an actual database.
One possible trick you might consider is to (manually!) add objects to the DB's schema that explicitly indicate its environment, e.g. `CREATE TABLE dbo.ThisIsProductionYouSillyPerson ( DummyCol int NOT NULL );` for prod and `CREATE TABLE dbo.ThisIsTestFeelFreeToMessAround ( DummyCol int NOT NULL );` - these tables would be excluded from the automated DB deployment code - and write test scripts that all start by checking that the `dbo.ThisIsTestFeelFreeToMessAround` table exists and that the `dbo.ThisIsProductionYouSillyPerson` table does not exist in the DB before continuing.
DB automation is great for preventing mistakes during common routine operations, but because DB automation can also go haywire and delete drop all-by-itself unless you set-up out-of-band (if that's the right term?) safeguards. Having airgapped dev-test-staging-and-prod won't help you if if you forgot the `WHERE` in an `UPDATE` in a little-used script that the prod automation uses, that testing never discovered (which happens all the time, it's scary).
I do appreciate how MySQL does come with an `UPDATE-without-key` guard, but I'm surprised none of the other RDBMS have safety-guards like that - just a simple `RequireManualConfirmationForMultiRowDml` flag on a table would help.
I had been there maybe 6 months. For some reason I can't recall I was meant to delete the staging RDS database.
Well, the databases weren't given human names, they were both long autogenerated strings.
I deleted the staging database and then prod stopped working
Whoops