[toomuchtodo]

No tech company is infallible. All of them have outages, some lasting hours, even days.

Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless.

One day it might just be your turn to break production.

[tptacek]

The subtext here isn't that Slack is bad at this (they are not), but that DNSSEC is somehow intrinsically unsafe (it probably is).

[toomuchtodo]

I agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS.

Perhaps I did not read the room appropriately. Mea culpa.

[Operyl]

Did Roblox finally come out with their postmortem blaming Consul? As far as I know we just assumed it, but have had no update since October.

[toomuchtodo]

https://news.ycombinator.com/item?id=29063026