[taviso]

Matthew, with all due respect, you don't know what you're talking about.

view-source:http://cc.bingj.com/cache.aspx?q=&d=4857656909960944&w=rj9cg...

view-source:http://cc.bingj.com/cache.aspx?q=&d=4901023173710126&w=n3mEZ...

view-source:http://cc.bingj.com/cache.aspx?q=&d=4558611265887320&w=urwoW...

view-source:http://cc.bingj.com/cache.aspx?q=&d=4592983872701813&w=Ghwdd...

view-source:http://cc.bingj.com/cache.aspx?q=&d=4997243316273666&w=wdpFH...

Not as simple as you thought?

[phaed]

Thousands of years from now, when biological life on this planet is all but extinct and superintelligent AI evolving at incomprehensible rates roam the planet, new pieces of the great PII pollution incident that CloudFlare vomited across the internet are still going to be discovered on a daily basis.

[mrich]

I was expecting this:

Thousands of years from now, when biological life on this planet is all but extinct and superintelligent AI evolving at incomprehensible rates roam the planet, taviso will still be finding 0-days impacting billions of machines on an hourly basis.

Be glad that Google is employing him and not some random intelligence agency.

[ar0]

I have huge respect for taviso and his team. Their track record in security work is so impressive. They are without a doubt extremely capable.

However, I am always wondering: are they really globally unique in their work and skill? So that they are really the ones finding all the security holes before anyone else does because they are just so much better (and/or with better infrastructure) than anyone else? Or is it more likely that on a global scale there are other teams who at least come close regarding skill and resources, but who are employed by actors less willing to share what they found?

I really do hope Tavis is a once-in-a-lifetime genius when it comes to vulnerability research!

[djsumdog]

One of the big conservatories in the infosec world are people who sell 0-day exploits to "security companies." Some go for the tens of thousands of dollars. Ranty Ben talked about how some people live off this type of income, when it came up in a panel discussion at Ruxcon 2012.

[tuyguntn]

No he is definitely not alone, some of them work for other security companies, for antivirus companies, some of them are selling found vulnerabilities

[kakarot]

What's funny is he kinda just stumbled upon this bug accidentally while making queries.

If I were just casually googling two weeks ago and came across a leaked cloudflare session in the middle of my search results I think I would have vomited all over my desk immediately. Dude must have been sweating bullets and trembling as he reached out on twitter for a contact, not knowing yet how bad this was or for just how long it's been going on.

[patcheudor]

Also still in Yahoo.

https://search.yahoo.com/search;_ylc=X3oDMTFiN25laTRvBF9TAzI...

http://208.71.46.190/search/srpcache?p=2001%3A56a%3Af651%3A6...

[patcheudor]

Bing and Yahoo with the same cached content. Interesting:

http://208.71.46.190/search/srpcache?p=yOGHqpbGWiXrRAIqLM87w...

http://cc.bingj.com/cache.aspx?q=%22X-SSL-Server-IP+104.16.5...

[npongratz]

I believe the 2009 Yahoo-Bing agreement is still in force, where Bing provides search results on Yahoo.com:

http://news.bbc.co.uk/2/hi/business/8174763.stm

I know the search I performed now on Yahoo states "Powered by Bing™" at the bottom.

[patcheudor]

Yeah, I thought that could be it as well but was at the bottom of the Yahoo result:

<!-- fe072.syc.search.gq1.yahoo.com Sat Feb 25 03:58:27 UTC 2017 -->

Given they are identical results it's pretty clear it must be a shared index I suppose, that or the leaked memory was cached.

[jsjohnst]

Yahoo provides a front end to the search results, Bing provides the crawl/search/archives.

[kakarot]

What the hell does Yahoo even do anymore? Just email? Or is that just a proxy to hotmail?

[detaro]

Isn't Yahoo search just a frontend to bing nowadays?

[dorianm]

Some IPv6 internal connections, some websocket connections to gateway.discord.gg, rewrite rules for fruityfifty.com's AMP pages, and some internal domain `prox96.39.187.9cf-connecting-ip.com`.

And some sketchy internal variables: `log_only_china`, `http_not_in_china`, `baidu_dns_test`, and `better_tor`.

[acqq]

Exactly, it looks that the cleaning people up to now only looked for the most obvious matches (just searching for the Cloudflare unique strings). There's surely more where "only" the user data are leaked and are still in the caches.

[tempz]

The event where one line of buggy code ('==' instead of '<=') creates global consequences, affecting millions, is great illustration of the perils of monoculture.

And monoculture is the elephant in the room most pretend not to see. The current engineering ideology (it is ideology, not technology) of sycophancy towards big and rich companies, and popular software stacks, is sickening.

[loup-vaillant]

How about clearing all the cache? (Or at least everything created the last few months.)

I've never seen anyone suggest it, I suppose It cannot or should not be done for some reason?

[tuyguntn]

You are asking for deleting petabytes of data. Some sides are interested in owning such data.

[mroi]

The real problem is going to be where history matters and you can't delete - for example archive.org and httparchive.org. There is no way to reproduce the content in the archive obviously, so no one will be deleting it. The only way is to start a massive (and I mean MASSIVE) sanitization project...

[baby]

or clearing all the cache of Cloudflares website. I think that's do-able.

[tuyguntn]

At this moment problem is not in Cloudflare's side, search engines crawled tons of data with leaked information, even though Cloudflare drops their caches, data is already in 3rd party servers (search engines, crawlers, agencies)

[fantyoon]

That's why he asked that the caches of all Cloudflare sites are dropped, not by Cloudflare but by these 3rd parties.

[jamaicahest]

That might work. If said 3rd parties were interested in helping. Most of them might be but it just takes one party refusing to help and then you've still got the data out there.

[baby]

no I meant, get a list of all domains using Cloudflare, get that removed from the cache of Crawlers.

[janwillemb]

Offtopic: "with all due respect" is often followed by words void of respect.

[Symbiote]

He is British. "With all due respect" means no respect is due. I don't think it's possible to show less respect while appearing polite. In other words, them's fighting words.

http://todayilearned.co.uk/2012/12/04/what-the-british-say-v...

[LeoPanthera]

This is perfectly fine if the amount of respect due is sufficiently low.

[77pt77]

Given the answers that cloudflare is giving I's say it's quickly approaching zero.

[janwillemb]

Ha! Excellent point!

[amenod]

Incredible. Are they really trying to pin it on Google? Yes, clearing cache would probably remove some part of the information from public sources. But you can never clear all cache world-wide. Nor can you rely that the part that was removed was really removed before being copied elsewhere.

The way I see it, time given by GZero was sufficient to close the loophole, it was not meant to give them chance to clear caches world-wide. They have a PR disaster on their hands, but blaming Google won't help with it.

[Bino]

You really have to see this to really grasp the severity of the bug.

[sah2ed]

The scope of this is unreal on so many levels.

20 hours since this post and these entries are still up ...

[sidcool]

Can anyone provide some context please ?

[Gigablah]

For anyone being linked directly to the post: the link back to the parent page is right on top: https://news.ycombinator.com/item?id=13718752

You can also click on "parent", and repeat as necessary.

[asdfaoeu]

The bottom of the file has contents from another connection. Notably

    HTTP/1.1
    Host gateway.discord.gg

[myth_buster]

Great(x3) parent https://news.ycombinator.com/item?id=13718752

[Gigablah]

After 16 hours, those cached pages are still up...

[nodesocket]

While it is good that you discovered leaked content is still out in the wild, your tone is somewhat condescending and rude. No need for it.

[dcposch]

You might not know the history here. Tavis works at Google and discovered the bug. He was extremely helpful and has gone out of his way to help Cloudflare do disaster mitigation, working long hours throughout last weekend and this week.

He discovered one of the worst private information leaks in the history of the internet, and for that, he won the highest reward in their bug bounty: a Cloudflare t-shirt.

They also tried to delay disclosure and wouldn't send him drafts of their disclosure blog post, which, when finally published, significantly downplayed the impact of the leak.

Now, here's the CEO of Cloudflare making it sound like Google was somehow being uncooperative, and also claiming that there's no more leaked private information in the Bing caches.

Wrong and wrong. I'd be annoyed, too.

--

Read the full timeline here: https://bugs.chromium.org/p/project-zero/issues/detail?id=11...

[baby]

I think this is a one-sided view of what really happened.

I can see a whole team at Cloudflare panicking, trying to solve the issue, trying to communicate with big crawlers trying to evict all of the bad cache they have while trying to craft a blogpost that would save them from a PR catastrophe.

All the while Taviso is just becoming more and more aggressive to get the story out there. 6 freaking days.

short timeline for disclosures are not fun.

[jgrahamc]

There was no panic. I was woken at 0126 UTC the day Tavis got in contact. The immediate priority was shut off the leak, but the larger impact was obvious.

Two questions came to mind: "how do we clean up search engine caches?" (Tavis helped with Google), and "has anyone actively exploited this in the past?"

Internally, I prioritized clean up because we knew that this would become public at some point and I felt we had a duty of care to clean up the mess to protect people.

[tlrobinson]

> "has anyone actively exploited this in the past?"

Has this question been answered yet?

[jgrahamc]

We're continuing to look for any evidence of exploitation. So far I've seen nothing to indicate exploitation.

[winteriscoming]

>> "has anyone actively exploited this in the past?"

Wouldn't your team now even have to decide how to deal with this even after some specific well known caches have been cleared? I mean there's no guarantee that someone may not have collected all this data and use it to target those cloudflare customer sites. Are you planning to ask all your customers to reset all their access credentials and other secrets?

[dcposch]

Google Project Zero has two standard disclosure deadlines: 90 days for normal 0days, and 7 days for vulnerabilities that are actively being exploited or otherwise already victimizing people.

There are very good reasons to enforce clear rules like this.

Cloudbleed obviously falls into the second category.

Legally, there's nothing stopping researchers from simply publishing a vulnerability as soon as they find it. The fact that they give the vendor a heads-up at all is a courtesy to the vendor and to their clients.

[baby]

> The fact that they give the vendor a heads-up at all is a courtesy to the vendor and to their clients.

It is the norm, and it is called responsible disclosure. You're trying to do the less harm, and the less harm is a combination between giving some time to the developers to develop a fix and getting the news out there for customers and customers of customers to be aware of the issue.

[empath75]

With all due respect, they should suffer a pr catastrophe.

[tedivm]

In this case I feel your comment is misdirected. Cloudflare was condescending in their own post above in which he was replying to- "I agree it's troubling that Google is taking so long" is a slap in the face to a team that has had to spend a week cleaning up a mess they didn't make. It is absolutely ridiculous that they are shitting on the team that discovered this bug in the first place, and to top it all off they're shitting all over the community as a whole while they downplay and walk the line between blatantly lying and just plan old misleading people.

[slipmagic]

I would be pretty mad if a website that I was supposed to trust with my data made an untrue statement about how something was taken care of, when it was not, and then publish details of the bug while cache it still out in the wild, and now exploitable by any hacker who was living under a rock during the past few months.

[nodesocket]

Actually I proxy two of my profitable startup frontend sites with CloudFlare, so I am affected (not really), but giving them the benefit of the doubt as they run a great service and these things happen.

[phaed]

They are well past deserving the benefit of the doubt.

I would also advise you notify your cloud-based services' customers how they might be affected (yes really), trust erosion tends to be contagious.

[josephb]

Agreed. The condescending downplaying tones displayed just aren't acceptable.

[nodesocket]

We only host our static corporate sites (not apps) and furthermore never used CF email obfuscation, server-side excludes or automatic https rewrites thus not vulnerable.

[potatosareok]

Hi,

I think you have misunderstood the issue. Just because YOU did not use those services does not mean your data was not leaked. It means that other peoples data was not leaked on YOUR site, but YOUR data could be leaked on other sites that were using these services.

[tptacek]

So far as I know, nothing like this thing has ever happened at any CDN ever before.

[rdl]

There have definitely been incidents where CDNs mixed up content (of the same type) between customers. Not exactly like this, but close.