[jssjr]

I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

[cddotdotslash]

I'm curious why you don't host your status page on a different domain/provider? When checking this AM why GitHub was down, I also couldn't reach the status page.

[ToastyMallows]

+1

The only way that I could check to see if Github knew they were having problems was by searching Google for "github status", and then seeing from the embedded Twitter section in the results page that there was a tweet about having problems. Twitter also being down for me didn't help the situation either.

[redbeard0x0a]

The attack is on the DNS servers, which take names like www.github.com and resolve them to ip addresses (i.e. 192.30.253.112 for me). Their status page is status.github.com - it is on the same domain name (github.com) as the rest of the site. Normally this isn't a problem because availability is usually something going on with a server, not DNS.

In this case, the servers (DNS server under attack at Dyn) that knows how to turn both www.github.com and status.github.com into an IP address were under attack and couldn't respond to a query. The only way to mitigate this would be to have a completely different domain (i.e. githubstatus.com) and host the DNS with a different company (i.e. not Dyn).

[cddotdotslash]

Right, this was my point. Hosting "status.domain.com" doesn't help much when it's "domain.com" that's having the problem. I think today's event will make a lot of companies consider this a bit more.

[Rapzid]

Hiiiinnnnndsiighhhttttt!!!!! Yeaaaahhhhyeahh!

Anyway, for them to take the github.com nameservers out of the mix they would need a completely separate domain name; would you know to look there?

You can delegate subdomains to other providers, but the NS records are still present in the servers listed in the registrar. So, you'd already need multiple DNS providers.. And you wouldn't have been down. Just sayin. I'm not sure anyone rated a DNS provider of this status getting hit this hard or completely as high enough risk to go through the trouble.

It's easy enough to look at a system and point out all the things you depend on as being a risk. The harder part is deciding which risks are high enough priority to address instead of all the other work to be done.

[mnordhoff]

I mean, some organizations do take precautions against this point of failure and use a separate status domain. Most don't.

https://www.dynstatus.com/ (using Route 53, at least today)

https://www.cloudflarestatus.com/ (using Dyn, ironically)

[jcl]

If it helps any, this link seems to work for me to reach the github status page (requires https certificate override, of course):

https://107.22.212.99/

[peterbonney]

Lots of companies use Twitter for that sort of real-time status reporting, whose own up/down status one would think is sufficiently uncorrelated... unfortunately the internet is complicated.

[joseluisq]

+1 Logical question!

[lanna]

This is what you can do to restore your GitHub access:

    grep github ~/.ssh/known_hosts
    sudo vim /etc/hosts
    sudo killall -HUP mDNSResponder
    ping github.com

[lukasm]

I added

192.30.253.112 github.com

but https://assets-cdn.github.com is failing

EDIT: Use 192.30.253.112 github.com 151.101.24.133 assets-cdn.github.com

or try 8.8.8.8 DNS

[lanna]

Why am I being downvoted for providing useful information? I don't understand HN...

[brown9-2]

Probably because you say to edit /etc/hosts but not what the content should be.

[tedunangst]

Is it hard to guess? The output of grep isn't a hint?

[PeCaN]

…except they did though, at least if you've sshed into github at some point (which I think nearly everyone has).

[oldmanjay]

If you're attempting to understand the behavior of individual users of HN as a collective, I can assure you that your initial principles are hampering you greatly.

[daenney]

Not sure if people aren't OK with the content but you've posted it twice, which is not really cool with most people or the guidelines.

Also probably the "hijacking top comment" part.

The other occurrence being here: https://news.ycombinator.com/item?id=12760156

[razster]

May not be HN doing the downvotes my friend.

[ionbara90]

seems like the right thing to do. however the ip address itself won't respond either

[afshinmeh]

Just being curious, why don't you use different DNS servers?

[toast0]

(I'm not Github, but I work for a Dyn customer) Using multiple DNS providers has technical and organizational issues.

From a technical perspective, if you're doing fancy DNS things like geo targetting, round robin though more A records than you'll return to a query, or healtchecks to fail out ips from your rotations, using multiple providers means they're likely to be out of sync, especially if the provider capabilities don't match. That may not be terrible, because some resolvers are going to cache DNS answers for way longer than the TTL and you have to deal with that anyway. You'll also have to think about what to do when an update applied successfully to one provider, but the second provider failed to apply the update.

From an organizational perspective, most enterprise DNS costs a bunch of money, with volume discounts, so paying for two services, each at half the volume, is going to be significantly more expensive than just one. And you have to deal with two enterprise sales teams bugging you to try their other products, asking for testimonials, etc, bleh.

Also, the enterprise DNS I shopped with all claimed they ran multiple distinct clusters, so they should be covered for software risks that come from shipping the same broken software to all servers and having them all fall over at the same time.

[pcai]

Most services, even if they aren't the size of Github, can't change their DNS provider on a dime.

[colanderman]

It's not a question of switching; you can host your DNS records at multiple providers.

[afshinmeh]

yup, that's what I meant. they can use different DNS providers, e.g. route53 AND dyn

[alberts00]

Route53 doesn't allow using it as slave DNS. https://forums.aws.amazon.com/thread.jspa?threadID=56011

[pyvpx]

more accurately, they don't support the common standard methodologies for transferring zone data between primary and secondary name servers (like NOTIFY, AXFR, etc).

there is nothing stopping you from having Route53 and $others as NS records for your domains. You just have to make sure they stay consistent. Apparently from the linked discussion, there are people offering scripts and services to do just that.

[snug]

Thats why you should have a different domainname

githubstatus.com instead of status.github.com

You could even through the domain on a free DNS service.

[stevekemp]

Maybe not, but you can store your records in a local place and push to both.

That's one of the reasons I setup a git -> Route53 setup at https://dns-api.com/

[3pt14159]

If this is consistently a problem why doesn't Github have fallback TLDs that use different DNS providers? Or even just code the site to work with static IPs. I tried the Github IP and it didn't load, but that could be for an unrelated issue.

[sly010]

> If this is consistently a problem why doesn't Github have fallback TLDs I don't believe this has been consistently a problem in the past. But after today big services probably will have fallback TLDs.

[jssjr]

Another status update from GitHub: "We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates."

We're maintaining yellow status for the foreseeable future while the changes to our NS records propagate. If you have the ability to flush caches for your resolver, this may help restore access.

Latest status message: https://twitter.com/githubstatus/status/789565863649304576

[BlackGuyCoding]

I love how the White House & GH posted a statement on Twitter.. that we can't access since its down.

[lambda]

Twitter's working fine for me. This attack is affecting different people differently; as a DDOS, attacking a distributed system (DNS) with a lot of redundancy, it's possible for some people to be affected badly while others not affected at all.

I briefly lost access to GitHub, but Twitter has been working fine every time I've checked. Posting status messages in multiple venues helps to ensure that even if one channel is down, people might be able to get status from another channel.

[JoshGlazebrook]

I wish you guys used statuspage or at least allowed email updates for the status of GitHub services.