[Animats]

    nslookup
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer.apple.com
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net", which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.

More:

    nslookup
    > developer-cdn.apple.com.akadns.net
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer-cdn.apple.com.akadns.net
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?

It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.

[lima]

> Or am I misreading that.

Yes:

    developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
    developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.

The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.

They fixed that and now it's back up.

This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).

> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.

[1]: https://ns1.com/blog/glue-records-and-dedicated-dns

[silisili]

> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.

It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.

The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.

[frays]

Works with other DNS servers.

  $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
  Server:  a.ns.apple.com
  Address: 17.253.200.1#53

  ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

  $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
  Server:  1.1.1.1
  Address: 1.1.1.1#53

  Non-authoritative answer:
  developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.201
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.202

[pul]

Now also works on their authoritative DNS servers again: https://www.nslookup.io/dns-records/apple.com#authoritative

[jonfw]

This looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well

[variant]

I think something with DNSSEC:

https://puck.nether.net/pipermail/outages-discussion/2022-Ma...

[joveian]

I noticed a few weeks ago that developer.apple.com was failing DNSSEC and that this had been going on for a while (follow the "previous analysis" links to see earlier errors as well):

https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/

It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.

lima has an anaylsis of the issue causing trouble:

https://news.ycombinator.com/item?id=30757487

[tptacek]

APPLE.COM isn't signed at all; this isn't a DNSSEC issue.

In the future, if you want to check if something is DNSSEC-signed (things rarely are: DNSSEC is overwhelmingly not enabled on the commercial Internet), you can just `host -t ds <domain>`.

[joveian]

I noticed it because developer.apple.com failed validation using systemd-resolved with DNSSEC enabled when someone posted a link on HN (but worked fine with DNSSEC disabled). It still does. The main apple site doesn't have that issue (the post I linked to gave the general, non-DNSSEC related issue this time).

I tried several local utilities and options but couldn't find a reliable way to determine if a site would resolve under systemd-resolved with DNSSEC enabled other than using systemd-resolve with DNSSEC enabled. It seemed like any time dnsviz.net shows an error the domain will not resolve, but some things it shows as warnings also cause sites to not resolve while other warnings do not. My favorite is that Verisign's DNSSEC validator's domain fails to resolve with DNSSEC enabled.

Possibly some or all of this is systemd-resolved doing the wrong thing, however the errors and warnings on dnsviz.net make me think this is not the case. www.google.com, for example, does not show any warnings or errors.

[tptacek]

GOOGLE.COM is also not DNSSEC-signed. Seriously, almost nothing is.

[joveian]

Right, but my point is "not DNSSEC-signed" does not seem to be the same as "free of configuration errors that prevent resolution of the name with DNSSEC enabled".

[tptacek]

AAPLIMG.COM isn't DNSSEC-signed either.

[variant]

Yeah. Was wondering if they'd enabled it and backed off when it didn't work.

[mnd999]

Can we refer to this as “Doing a Facebook?”

[Karrot_Kream]

This has nothing to do with the BGP failures that FB had earlier. This is a DNS configuration problem. It's much simpler to fix.