[WestCoastJustin]

Here are the makings of a bad week (Monday of all things)

- MySQL schema migration causes high load, automated HA solution causes cascading database failure

- MySQL cluster becomes out of sync

- HA solution segfaults

- Redis and MySQL become out of sync

- Incorrect users have access to private repositories!

Cleanup and recovery takes time, all I can say is, I'm glad it was not me who had that mess to clean up. I'm sure they are still working on it too!

This brings to mind some my bad days.. OOM killer decides your Sybase database is using too much memory. Hardware error on DRBD master causes silent data corruption (this took a lot of recovery time on TBs of data). I've been bitten by the MySQL master/slave become out of sync. That is a bad place to be in.. do you copy your master database to the slaves.. that takes a long time even of a fast network.

[cageface]

This kind of thing is one of the main reasons I prefer to do app development instead of backend work now. I don't get calls at 3am any more.