Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net",
which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.
More:
nslookup
> developer-cdn.apple.com.akadns.net
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer-cdn.apple.com.akadns.net
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point
"developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?
It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.
developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.
They fixed that and now it's back up.
This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).
> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.
> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.
It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.
The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.
This looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well
I noticed a few weeks ago that developer.apple.com was failing DNSSEC and that this had been going on for a while (follow the "previous analysis" links to see earlier errors as well):
It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.
lima has an anaylsis of the issue causing trouble:
APPLE.COM isn't signed at all; this isn't a DNSSEC issue.
In the future, if you want to check if something is DNSSEC-signed (things rarely are: DNSSEC is overwhelmingly not enabled on the commercial Internet), you can just `host -t ds <domain>`.
I noticed it because developer.apple.com failed validation using systemd-resolved with DNSSEC enabled when someone posted a link on HN (but worked fine with DNSSEC disabled). It still does. The main apple site doesn't have that issue (the post I linked to gave the general, non-DNSSEC related issue this time).
I tried several local utilities and options but couldn't find a reliable way to determine if a site would resolve under systemd-resolved with DNSSEC enabled other than using systemd-resolve with DNSSEC enabled. It seemed like any time dnsviz.net shows an error the domain will not resolve, but some things it shows as warnings also cause sites to not resolve while other warnings do not. My favorite is that Verisign's DNSSEC validator's domain fails to resolve with DNSSEC enabled.
Possibly some or all of this is systemd-resolved doing the wrong thing, however the errors and warnings on dnsviz.net make me think this is not the case. www.google.com, for example, does not show any warnings or errors.
I always use Apple Maps, but once in a while if I'm in an unfamiliar city and the Apple Maps directions seem suspiciously weird, it is useful to have Google Maps app for a sanity check. (directions to a particular pier at the Seattle waterfront were insanely incorrect via Apple)
On my side in France apple Map only partially work. Basemap are displaying correctly but query and routing function are unreachables. "Domain name not found" (translated from french). So it could be a DNS meltdown?
Usually basemap because they are heavy are served through a separate CDN.
Probably true, but we wanted it for driving directions via CarPlay and were in a bit of a rush. The car's built-in navigation (which we otherwise never use) ended up working fine, but the browser versions probably would have been my next attempt.
I’m sure it wasn’t when you posted 10 minutes prior, but FWIW currently listing 11 outages:
> App Store - Outage
Today, 12:32 PM - ongoing
Some users are affected
Users may be experiencing intermittent issues with this service.
Apple Arcade - Outage
Today, 12:32 PM - ongoing
Some users are affected
This service may be slow or unavailable.
Apple Music - Outage
Today, 12:32 PM - ongoing
Some users are affected
This service may be slow or unavailable.
Apple TV+ - Outage
Today, 12:32 PM - ongoing
Some users are affected
Users may be experiencing a problem with Apple TV+. We are investigating this issue.
iTunes Store - Outage
Today, 12:32 PM - ongoing
Some users are affected
This service may be slow or unavailable.
Podcasts - Outage
Today, 12:32 PM - ongoing
Some users are affected
Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.
Radio - Outage
Today, 12:32 PM - ongoing
Some users are affected
This service may be slow or unavailable.
Apple Business Manager - Outage
Today, 1:14 PM - ongoing
Some users are affected
Users may be unable to sign in.
Apple School Manager - Outage
Today, 1:14 PM - ongoing
Some users are affected
Users may be unable to sign in.
Device Enrollment Program - Outage
Today, 1:14 PM - ongoing
Some users are affected
Users are experiencing a problem with this service. We are investigating this issue.
Schoolwork - Outage
Today, 1:14 PM - ongoing
Some users are affected
This service may be slow or unavailable.
A lot of system status pages are updated by humans who will verify issues before reporting them. Main reason is to avoid overly surface every minor and transitory issue to public view.
It's very easy, except when it's hard. Also, it's never easy.
Joking, but only somewhat. That's because the easy cases are handled by automation, etc. If you knew it could happen, you probably planned for it. Figuring out what the issue is, if there really is an issue, and the scope of the issue can take some time.
No. “Doesn’t respond for me” doesn’t imply “down for lots of people”. If you discover that foo.com doesn’t respond, it takes a while to figure out whether that’s on your system, in your network, in the city, etc.
Yes, you would set up multiple hosts across the world polling that server, but that adds complexity. Maybe, those pollers decide the site is down because of a bug in your network setup, while the rest of the world happily uses your services.
Pingdom seem to manage it. Pretty sure one of the FAANGS could to. I appreciate an obscure managed service might be a bit diffcult, but main developer site?
My response was to “Quite easy to verify if the entire developer site is down though, non?”
I never claimed it’s impossible, just that it isn’t “quite easy”, especially to check that the “entire developer site is down”. The home page may be down, with the rest being up, the home page may be up, with the rest being down, etc.
I chose the perfect time to restore a repaired iPhone, don’t seem to be able to fully login to iCloud, it’s hanging on the login screen…
Edit:
It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.
Edit:
Login and app downloads now working as of 6.00GMT
It's times like this that force us to remind ourselves how reliant we are on critical services like these. On one hand, we can celebrate (Internet snow-day!) but on the other we are forced to shop around for alternatives too.
I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!
I’m sure it’s happens more than I’m aware but i have to say that i can’t recall an App Store outage since i got back in the platform 3-4 years ago. Not bad!
Definitely. Downdetector shows a bunch of reports too (e.g. https://downdetector.com/status/apple-music/). I noticed issues with Music and News, seems like a ton of their services are down
They're fine for knowing that something is going on, but not great for knowing exactly what the cause is.
For example, when Facebook's services went down in October, people were reporting that AT&T and other cell carriers were down because they couldn't open the apps. As far as I know there wasn't an outage with any of the carriers that day.
I think they’re about as useful as any anecdotal data out there. Unusually high numbers of reports when you’re seeing issues yourself is about as good as it gets until a status page is updated (which it thankfully has been finally).
Would an infinite spinner also show up if the server was up but the connection was problematic? If yes, this would be about not handling network errors, which sounds like a decent rejection reason to me.
iCloud Private Relay is shown as affected as well. This is an interesting case when it comes to failure behavior. From security perspective, you want your connection to stop working instead of falling back to insecure. Is this the case? Can anyone confirm?
It fell back to insecure for me, for about 30 seconds (maybe longer before I noticed) I couldn’t connect to the Internet from my iPhone, then I got a notification saying private relay was unavailable and I was able to connect again.
A few minutes later it gave me another notification saying private relay was working again.
They seem to have been having a bit of a lie-down, today. I can't submit TestFlight builds, but now, it is taking longer, before the server throws a nutty, so I guess the fix is on its way.
the domain name developer.apple.com resolves through a series of CNAMEs to Apple's CDN (applimg.com), which if it was down would explain other things like iMessage also being unavailable.
If I had just dropped $2K-$12K on a media-centric computer with the intent of running encrypted backups, spreadsheets, databases and other inappropriate tasks for non-ECC memory (looking at you Leo), I’d downvote too!
More:
It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.