[toast0]

Have you worked with redundant routers? They certainly reduce the number of outages, but sometimes the hardware (or software) fails in exciting ways that doesn't engage the redundancy, or doesn't engage it properly, and you still get an outage (or you get an outage that wouldn't have happened). Or sometimes, one circuit is out of service for repair or upgrade, and the other circuit is connected to the router that failed. And routing for the AS that travels on that circuit was set not to fallback to transit because the last time that happened, it caused major issues.

I have no specific knowledge of today's events, but this sort of thing happens. You can get the number of incidents down pretty low, but not to zero.

[nineteen999]

I remember seeing a Security Analyst for the DC I worked for take down 6 racks worth of Cisco Catalyst 12000 series router hardware once.

They had a HSRP interface set up at the .1 address, and the security analyst set his laptop up with the same static .1 IP address and plugged it in. Instant outage.

[ocdtrekkie]

I have. I am just highlighting that the problem surely should be more complex than described. Or that their redundancy for these events was not adequately devised.

[toast0]

Google often releases a pretty solid post-mortem, which will give the detail of the event. The level of detail appropriate for same-day release is really 'router failure' or 'power failure' or 'software failure' or 'vehicle drove into the building failure'. Expecting more than 'we know what it was, and we fixed it' or 'we don't know what it was, but it stopped happening' or 'yes, we're working on it' on a same-day twitter post is unreasonable.