[achiang]

Google networking SRE here (my team runs ns[1-4].google.com among other services).

Regardless of original intent, the blog doesn't land well with me. It could have provided the background on flowspec, using their own past outage as a case study, without any of the speculation or blameyness that came across here. The #hugops at the end reads quite disingenuously.

We see other networks break all the time and we often have pretty good guesses as to why. But I personally would never sign off on a public blog speculating on a WAG of why someone else's network went down. That's uncouth.

[_8j50]

I think you're stuck on the politics. Level3 is their competition but initially CF was blamed. CF owes it to their customers and investors to explain to them why they had an outage and how they responded to it, and they do not need talk in detail about an unrelated past incident (just because it was related to flowspec does not mean it was a similar outage), and they certainly should not wait for Level3's investigation.

I would expect Google to have a similar explanation if a significant number of GCP customers faced an outage.

You should know, it wasn't just someone else's network that went down, that network brought down a big chunk of the internet with it. I think technical honesty comes before political appearances. The #hugops and mention of their past experience with a flowspec outage is clearly there to signal that the blogpost is not there for blaming or making L3 look bad.

[achiang]

The politics is exactly the point of my comment.

The professional way to write a blog post like this is from your own perspective. Identify the proximate cause (the peer), name names if you must, talk about how awesome your own systems are, show some of your monitoring if you like, and talk about what you'll do in the future to be even more resilient to this class of problems.

That's all to the good and much of Cloudflare's blog was exactly that. Would've been fine if they left it like that.

Acknowledging there is no postmortem (yet) but then pointlessly speculating about what it might contain is what I have a problem with.

I don't speak for Google but if I found out we had written a post like this, I would speak up and advocate to change it.

[_8j50]

There is nothing professional about avoiding a topic for the sake of appearances. Level3 put out details knowing others in the industry will discuss and speculate based on that information. They could have witheld details such as flowspec and edge routers bouncing but they did not, it's perfectly professional to discuss speculative details of someone elses outage that affected your customers based on details they chose to make public.

In infosec for example, it's extremely common to speculate about a vulnerability based on details in the CVE. Entire news articles are based on such speculation. Like I said, you are giving too much weight to optics and appearances. I would like to see anyone actually at Level3 complain about this post.

[timcosta]

Honestly, I'm wondering if this blog is a response to https://web.archive.org/web/20200830171114/https://www.cnn.c... which has since been heavily modified, but was on the front page of CNN making it sound like Cloudflare was responsible if you only read the first bit.

[_8j50]

Could be, even yesterday the BBC was blaming CF at the text scroll thingy on their newscast

[iJohnDoe]

Normally CF throws a lot of mud in these situations. Karma got them on their recent last outage.

While reading this I got impression they were genuinely trying to tone down the mud slinging they normally do while also trying to make it clear the outage wasn’t their fault. They just need more practice.

A provider as large as Cloudflare will always be impacted by other providers. Hopefully that point is clear to them now. The worst thing that can happen is they get a reputation they can’t play ball nicely and their peers and partners get tired of them. Service to their customers will erode over time because those peers and partners will screw with CF behind the scenes. It’s better to have friends than enemies with the type of business CF is in.

[Serow225]

pretty common pattern for CF though.