IO domain DNS failure

[yoshiokatsuneo]

For 3 or 4 hours, IO domain DNS lookup often fails. Now, only 2 out of 7 IO domain nameservers respond, and other 5 DNS nameservers are dead.

[xiaodown]

AWS response:

Hello,

Some customers have reported intermittent resolution issues with .io domains. We can confirm that Route 53 DNS services are operating normally at this time and these issues seem to be related to the .io top-level domain provider.

We've investigated the issue and are primarily seeing incorrect responses from two of the .io nameservers: ns-a4.io and ns-a2.io. These nameservers are returning NXDOMAIN intermittently for domains that do exist. As a result, once a resolver receives the erroneous response, it will cache the non-existence for the negative TTL, which for .io is set to 3600 seconds (1 hour).

One suggestion we have is to increase the TTLs for your domains in Route 53 so that the resolvers cache DNS answers for a longer duration.

Best regards,

(Customer support rep name redacted)

Amazon Web Services

We value your feedback. Please rate my response using the link below.

[JonAtkinson]

I recieved this reply from nic.io support a few minutes ago: http://imgur.com/a/wV2Kk

[a10c]

I have been seeing the same failures for many IO TLDs

https://www.whatsmydns.net/#A/gitlab.io

Example: https://i.imgur.com/SU4BAHK.png

I have noticed the failures occurring moreso in Asia and Russia than in other regions.

[yoshiokatsuneo]

So, "ns-a4.io" is the culprit! On ns-a4.io, xxx.io can be resolved, but yyy.xxx.io cannot be resolved(delegated). Ref: http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39

[mic159]

We are also having issues resolving some .io domains from AWS Sydney (ap-southeast-2)

[matt_oriordan]

For those interested, we wrote a blog post about single points of failure with DNS last Friday when the Dyn attack happened. Our strategy ensures our customers are mostly unaffected by issues like this. See https://blog.ably.io/routing-around-single-point-of-failure-...

[yoshiokatsuneo]

There are 7 IO nameservers (ns-d1.io, ns-l1.io ,ns-y1.io, ns-a1.io, ns-a2.io, ns-a3.io). Only ns-a4.io and ns-a3.io respond.

[yoshiokatsuneo]

It looks, the the problematic nameserver "ns-a4.io" is down. So, the problem should be disappeared...

At the same time, it looks, there is only one "ns-a3.io" nameserver working for IO domain, at now...

[xiaodown]

Yep, rather than no such domain or a bad delegation, the "bad" servers seem to just be down now.

[jansh]

For Planio (https://plan.io/), all accounts are available via yourname.planio.net as an alternative while this outage lasts.

[tkazec]

We've been seeing this for exactly five hours now, all across East Asia.

Edit: Looks like .io is managed in London, and their customer service off hours match the downtime.

[alwaysanon]

Two of their servers were sending out NXDOMAINs which are then cached by downstream forwarders. It seems they have turned those off now.

[matt_oriordan]

This is still affecting us. Our customers on Heroku seem to be the worst affected as Heroku is unable to resolve the DNS

[asher_]

We are experiencing the same from Vietnam

[lifeeth_]

.sh and .ac are also run by the same folks - looks like there are issues with those tlds as well.

[user9999]

Having the same issue in Singapore. got nothing to do, helpless.

[yjchang]

Same here in South Korea. Still not fixed.

[xiaodown]

Getting significantly worse now.