Not only Copilot, seems like some Microsoft services like Bing AI and Bing Image Creator have some issues today as well with 4xx / 5xx, and incorrect region authorization (had to switch the account region from a European country to US to make it work again on mobile)
I think you may have missed the joke - they were implying Github was using Copilot internally, causing the outages, due to poor output. Not that Copilot itself was unavailable (although that may be true, also)
They blamed the march and april outages on some database query that was changed due to an infrastructure change they rolled out. I'm guessing their infrastructure change caused some other race condition issue that they are only seeing after major production failure due to not load testing enough in their staging environment https://github.blog/2023-05-03-github-availability-report-ap...
Sales handing out Office365 discounts and trying to convince people that AWS and GCP is going to steal their data, judging by companies I worked for that used Azure.
Wasnt it true? Thas Amazon abused their AWS position and stole their competitors data, so thats why Germany's retail businesses are building their own Clouds
I worked on the volume licensing part of Microsoft years ago and deployments were stressful. They'd start at friday late like 8pm or so and go until 8am in the morning. Everyone was on a long call the entire time. I hated it.
Today seems worse than yesterday. I'm getting wildly inconsistent results when viewing repositories after a push. Hard to tell if my push actually went through, and it's not triggering actions.
From an SRE, one of their DB clusters failed. They use Vitess which is great, but it can be prone to hotspots and doesn't auto-shard. Heavy usage (esp. from large customers, rogue jobs) can take down the cluster. When it goes down, it's a PITA to resolve.
Ah, unbalanced shards via wrong sharding keys was an issue at one point, IIRC. I remember talking with an SRE there when something bad happened at GitHub last year, and I know that this time the current DB cluster failed.
To be clear, I _was_ mapping previous incidents with this year's incident — no competitor or hard feelings involved. I really like Vitess, fwiw. And the only thing I really love is FoundationDB :)
Side note: "Autosharding" is largely a myth that unproven databases are touting. Sharding is complex and requires planning and control. Databases that start shuffling data round without oversight produce nasty surprises. Trying to be too magic is normally always a mistake with databases.