[poooogles]

I've always wondered how Google would deal with a client on GCP being DDoSed. Mainly as were in online advertising and DDoS extortion isn't uncommon.

Guess with this I'll now find out, as crapping on Krebs' site is practically a right of passage when you've got a botnet now.

[hrrsn]

I'd wager this is Project Shield rather than regular GCP. https://projectshield.withgoogle.com/public/

[pfg]

There's a "server: shield" response header for krebsonsecurity.com, so this is probably a good guess.

[Springtime]

Krebs confirmed this in his latest blog post [1].

Today, I am happy to report that the site is back up — this time under Project Shield, a free program run by Google to help protect journalists from online censorship.

[1] https://krebsonsecurity.com/2016/09/the-democratization-of-c...

[eric_bullington]

This is the first I've heard of Google's Project Shield. Very cool project, is it relatively new? Does anyone know if this was the result of someone's 20% time or a top-down Google initiative?

[kerneis]

Project Shield is part of Jigsaw (formerly "Google Ideas"): see https://jigsaw.google.com/ and https://medium.com/jigsaw for more related projects and context.

Google Ideas is 5 years old now. I'm not aware of it having started as a 20% project. [disclaimer: I work at Google, but not on Jigsaw.]

[r3bl]

I'm working for a journalistic organization that cooperated with Google to create one of the graduated Jigsaw projects (Investigative Dashboard).

We were considering using Project Shield relatively recently (as in, some time this year), but at that time, the info about how it actually worked was... well, non-existing on their website. IIRC, their SSL support is also fairly new, which made it pretty unusable until very recently.

[boulos]

Interestingly, I've never looked deeply at what Project Shield does under the covers but I know what all the underlying technology "must" be. The main difference between GCLB and Shield, is that Shield is a free service (operated by a different group, as already mentioned) explicitly for those at risk of censorship.

As another poster indicated, if you want someone to terminate SSL for you, you're going to need to hand them a key. We encrypt ours at rest, provide the same security and care to your secret material as we do Google properties, and as you can see with our Customer Supplied Encryption Key support for Persistent Disk and GCS, we care a lot about letting you control access to your data. If you don't mind me asking, to whom are you comfortable uploading your keys to?

Disclosure: I work on Google Cloud, so I'm actively trying to take your money in exchange for our services.

[mschuster91]

> As another poster indicated, if you want someone to terminate SSL for you, you're going to need to hand them a key.

Didn't Cloudflare invent Keyless SSL to solve this problem? https://www.cloudflare.com/keyless-ssl/

[eli]

Well by design it's putting Google in a MITM position for your site (as is Cloudflare for the matter). You're either OK with that or you're not.

I don't see how they'd be able to effectively stop Layer 7 attacks without being able to see unencrypted requests.

[robin_reala]

Interestingly their FAQ says:

Project Shield welcomes applications from websites serving news, human rights, or elections monitoring content. We do not provide service to other types of content, including gaming, businesses, or individual blogs.

I guess Brian’s a reasonable exception.

[jpgvm]

Further down they explicitly mention journalists, which I believe is a reasonable category for Krebs.

[boulos]

As I said down thread, we (Google) and some of our customers are routinely the target of attacks (DDoS and otherwise). We don't talk about it (sorry? yet?), but we have mechanisms to protect you as well as ourselves and other customers. My hope is we'll be talking about a particularly high profile one soon, but if you've got an account and want to talk to us privately, we're here to help.

Disclosure: I work on Cloud, but I'm not a networking expert.

[jaypaulynice]

The underlying technology isn't that difficult to understand...there is a web application firewall that sits in front of the web server serving the real content. There is also a load balancer to distribute the requests.

You can block the attack based on IP, request headers, rate control, geo, etc. The problem with some ddos attacks is the size of the request body. That's what usually overwhelms a server.

Ddos attacks do knock out the servers, but since the requests are distributed, then you don't see that. Because so many servers are getting knocked out, then real paying customers can't serve their content. I'm guessing that's why they pulled the plug.

Google's infrastructure is so large that knocking out a few servers won't impact much, they just spin them back up.

[pktgen]

I realize Krebs seems to be using Shield, not vanilla GCP, but I'd like to know the answer to this as well.

Krebs' article describes two attack methods, fake HTTP requests and a simple volumetric flood of GRE packets. L7 mitigation clearly isn't part of GCP's service. The GRE packets should be a different story -- if the customer isn't using GRE, they should be able to drop this traffic easily using the GCE firewall or GCE load balancer.

The vulnerability report at http://www.securityfocus.com/archive/1/532239 suggests the GCE firewall is, or at least was as of May 2014, ineffective against such an attack (even if your firewall rule set will drop the problem traffic). But it also says the GCE load balancers do not have the same problem. Krebs didn't mention how much of the 600+ Gbps was GRE, but let's say a GCE load balancer was configured to pass through only TCP/80 and received 100 Gbps of non-TCP traffic. Would the load balancer function as intended and drop all of this with no interruption to legitimate TCP traffic, would Google null route you, or something else?

[poooogles]

I fear it would be null routing. Would be good to know someone with some experience of this that's not from Google.

We can get all the reassurance from an account manager that they'd never stop our service. Real life doesn't always play out that way though...

[boulos]

I don't think that's true for GCLB. We might rate limit packets arriving or some such, but I'm not a networking expect (as with other posts, I'm sure someone else at Google will send me an excellent email explaining a subtlety here).

We generally don't talk about DoS attacks at Google (sorry). But I can say we're familiar with them, and that Cloud uses the same front end and networking infrastructure as the rest of Google here. We do understand (per project, the unit of billing) if there's tons of ingress/egress from a single project, and your intuition is right: it would be wise to respond to "Hey, is this 1 Tbps of traffic legitimate?" quickly ;).

Disclosure: I'm an engineer in Cloud (mostly GCE), but since I'm not an account manager I refuse to reassure you ;).

[pktgen]

Thank you for providing what information you can.

I'm a game server operator. I'm not currently running anything, but I probably will in the future. We used to receive volumetric L3/L4 floods all the time from booters. The good thing is these were always just dumb kids, not sophisticated attackers, so they would be easy to filter with a simple ACL (often they were targeting some random port we don't use, so a simple rule set of "accept whatever UDP and/or TCP ports we use, drop all else" would be effective). There were a few providers who would set up these ACLs, but they were only effective up to a few Gbps and would null route past that.

If I get back into this, I will give GCE a try with GCLB in front for, effectively, ACLs. In my experience it's a near certainty that any reasonably popular game server will be attacked (very often as retaliation for banning someone for cheating). I'd imagine your mitigation capabilities for a small-time game server customer paying a few hundred a month would be different than a large corporate client paying 5 figures, but I'd still like to try GCE for this as it seems like the best bet (besides OVH).

[roel_v]

*rite

(sorry)

[poooogles]

TIL. Never knew that thanks.

[qoqiwd8o1i2j]

rite of passage, rite of spring ...

[MaysonL]

Trying to access 130.211.45.45, which is listed in OP as the IP adress, keep getting 502 server errors.

[poooogles]

Sure you're sending the right host header?

[neitsab]

You can try hardcoding the IP address in your hosts file (see my comment below: https://news.ycombinator.com/item?id=12574428#12575021). This will get you to the site through a regular request.