Hacker News new | ask | show | jobs
by chrislomax 4517 days ago
Simple one really and probably most common. Realise a data integrity issue on DB, try to load from backups and notice that the backups have the same integrity issue. Find a backup from about 2 weeks previous where data is intact and piece together the good pages from the daily backups and the 2 weeks old one.

All in, took 4 days and a new server where the hard drive had stored bad pages on the DB. We lost 2 days of orders (they were processed through to the internal systems though so not really lost)

Lesson learned, validate backups and check page integrity when backing up