[tptacek]

Holy shit. That's bad.

What this suggests is that Slack, for reasons passing understanding, enabled DNSSEC on their zones (with a DS record that essentially turns DNSSEC on, and the accompanying key records) --- then disabled DNSSEC by pulling all the records. But the DS records are in caches; validating resolvers go looking for the keys, which don't exist, and say "welp, I guess Slack.com doesn't exist".

[aeden]

I wonder if they are using tooling that doesn't properly retain DNSKEY records for DS that recently removed? This is one of the reasons we perform controlled automated key rotation and removal in DNSimple, so that we can ensure we retain the keys in the authoritative zone on each key rollover giving the DS records time to expire from caches.

[icedchai]

It was especially bad since their status page wouldn't even resolve! I eventually just restarted my local caching DNS server.

[ithkuil]

We had a DNS related outage with route53. Some of our zones just lost some records and then they reappeared. Could that explain what happened to slack's DNSSEC related records?

[terom]

A good question, and apparently enough to elict a response: No!

> This issue was caused by our own change and not related to any third-party DNS software and services.

[tialaramex]

Aren't you, in fact, the same Thomas Ptacek who has repeatedly claimed that DNSSEC is so irrelevant that events like this would go essentially unnoticed?

Edited to add, e.g. https://news.ycombinator.com/item?id=22400167

> DNSSEC is moribund and almost nobody uses it; in reality, the DNSSEC root private keys could land on Pastebin tomorrow and nothing would "break"

We have this whole thread here about a "service disruption" for Slack, and nobody leaked the "root private keys" just one person made a dumb error and it blew up their site.

[tptacek]

No, I'm the Thomas Ptacek who has repeatedly claimed that the only impact DNSSEC is going to have on the Internet is causing outages like this. It's right there in the blog posts; in fact, it's even in the 2007 blog posts I wrote about this on the Matasano blog.

[ptomato]

> just one person made a dumb error and it blew up their site

yeah, the dumb error they made was "using DNSSEC"

[bluejekyll]

I'm not going to defend DNSSEC here, because this outage and others continue to support tptacek's perspective on its usefulness.

But, some governments are requiring DNSSEC, which regardless of its usefulness, puts companies that want those contracts in a bit of a bind.

Perhaps it would make sense to split domains such that DNSSEC guarded ones would not negatively impact ones that do not have DNSSEC.

[tptacek]

The USG DNSSEC requirements, which seem to be a part of what happened, are fragmented and incoherent. OMB withdrew DNSSEC requirements in 2018, and CLOUD.GOV doesn't support it. But some older requirements documents still have them, and need to be updated.

The important top-line thing to know here is that virtually all tech companies eschew DNSSEC (you can verify that for yourself with `host -t ds stripe.com`; substitute any other company for Stripe.

DNSSEC-quarantine TLDs are a good idea.

[acdha]

If anyone else is curious about the OMB cycle, here's a pretty good explanation with links to the source memos:

https://cloud.gov/docs/compliance/domain-standards/#dnssec

[terom]

The rationale in OMB memo M-18-23 for withdrawing the DNSSEC requirement in M-08-23 doesn't seem very convincing: we don't need this anymore because everyone should already have DNSSEC by now?

> M-08-23, Securing the Federal Government's Domain Name System Infrastructure (August 22, 2008)

> OMB is rescinding memorandum M-08-23, which provides additional guidance on the Domain Name System (DNS), specifically focusing on new security protections for the Federal DNS. The requirements in this memorandum are outdated; agencies already should have implemented these security protections.