| One of the most satisfying feature degradation steps I did with FastComments was making it so that if the DB went offline completely, the app would still function: 1. It auto restarts all workers in the cluster in "maintenance mode". 2. A "maintenance mode" message shows on the homepage. 3. The top 100 pages by comment volume will still render their comment threads, as a job on each edge node recalculates and stores this on disk periodically. 4. Logging in is disabled. 5. All db calls to the driver are stubbed out with mocks to prevent crashes. 6. Comments can still be posted and are added into an on-disk queue on each edge node. 7. When the system is back online the queue is processed (and stuff checked for spam etc like normal). It's not perfect but it means in a lot of cases I can completely turn off the DB for a few minutes without panic. I haven't had to use it in over a year, though, and the DB doesn't really go down. But useful for upgrades. built it on my couch during a Jurassic park marathon :P |