|
|
|
|
|
by velcrovan
842 days ago
|
|
Seems like whenever I log into FB lately it's pretty much always in a state of “graceful feature degradation”. For example, as soon as I log in I see a bell icon in the upper right with a bright red circle containing an exact positive integer number of notifications. It practically screams “click here, you have urgent business”. I can then leave the web page sitting there for any number of minutes, and no matter how long I wait, if I click on that notification icon it will take a good 20 seconds to load the list of new notifications. (This is on gigabit fiber in a major metro area, so not a plumbing issue.) |
|
To give a super simple example, suppose you have a database that can transparently fail over to a backup, but it's so "transparent" that nobody even gets notified. Suppose the team even tests it and it proves to work well. The team will then believe that they are very well protected and tell all their customers and management all about how bulletproof their setup is, but if they don't notice that the primary database corrupted and permanently went down in month six because their systems just handle it so well, they'll actually be operating on a single database after all and just be one hiccup from failure.
One of the jobs of an ethical engineer is to make sure management doesn't just say "it's OK, the site is working, forget about it and work on something else" without some appropriate amount of pushback, which you can ground on the fact that sure, they're saying to ignore it now, but when the second DB goes down and the site goes down they sure won't be defending you with "oh, but I told the engineering team to ignore the alerts and keep delivering features so it's really my fault and not theirs the site went down".
At Facebook's scale, something will always be in a state of degradation. It's just a fact of life.