Hacker News new | ask | show | jobs
by terramex 1682 days ago
Etsy is 404 too: https://www.etsy.com

Seems to be a bigger issue.

edit: Nest is down too: http://nest.com

Fitbit.com is 404 too: https://www.fitbit.com

Big GCP issue?

edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League.

GCP status page is still green all over the board: https://status.cloud.google.com

19:10 CET update: Some websites are coming back, including spotify.com, but their app still does not work for me.

information about outage just added to GCP status page, direct link: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-11-16 09:53 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-11-16 10:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

19:20 CET update:

Description: We believe the issue with Cloud Networking is partially resolved.

Customers will be unable to apply changes to their load balancers until the issue is fully resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2021-11-16 11:28 US/Pacific with current details.

Spotify desktop app still not working for me.

19:45 CET: Spotify app is back online for me.

14 comments

Another cloud issue? Wow, nearly a week's uptime. https://news.ycombinator.com/item?id=29197832
When Spotify went down I switched to Youtube Music. Funny how that's still running but Spotify isn't, if it is in fact a Google issue.
Internal Google infra and GCP are different beasts. Sure some failure modes take both down but more likely than not they are independent.
That's why many choose AWS, they eat their own dog food.
Parts of google are also down it seems, if one clicks three lines on the top right on google.com in mobile browser, then the https://www.google.com/mobile/?newwindow=1 opens, which also appears 404 not found.
I’m assuming all these websites point to Google web frontend servers and for some reason it’s no longer able to map the Host header to the proper backend to proxy to.
This is a sort of repetitive outage for Google. They've wiped out the GSLB configs before. A year ago there was also that big outage where they blanked out the whole Gmail delivery configs and started rejecting all mails (even for gmail.com). Config safety is not their strong suit.
For what it's worth, not only HTTP(S) load balancers went down. We have a couple TCP proxy load balancers that went down too.
I was seeing regular levels of health checks throughout the entire outage, so they still had configs in place. That seems plausible.
https://deno.land down too. I guess a lot of CI actions will be failing just like ours...
deno.land is working for me, while lots of other sites are not. Did they failover to another provider?
Working now, yes. Though deno.land's current IP still nslookup's to something inside googleusercontent.com so maybe google has fixed their Side. But it was definitely offline earlier today (cf. our github CI failures and downforeveryoneorjustme)
It seems things are gradually coming back. All of our load balancers started working again.
discord seems to be down as well
homedepot.com is 404, various pages on lowes.com are 404
Egnyte had a pretty major outage today too.

https://status.egnyte.com

I have things running on GCP that are OK. I think it's related to nameservers.
I don't think so, https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ shows that spotify.com resolved to the same 35.186.224.25 as it does now.
It is (was?) definitely a load balancer issue. We had multiple load balancers on completely different DNS providers fail.
All work for me in the EU apart from Etsy.
Homedepot.com appears down as well.
Gojek and Tokopedia too.
Evernote too.