[gamegod]

Irony alert:

> "But technology providers in the United States could suffer blowback. As Dyn fell under recurring attacks on Friday, Mr. York, the chief strategist, said such assaults were the reason so many companies are pushing at least parts of their infrastructure to cloud computing networks, to decentralize their systems and make them harder to attack."

Pushing your infrastructure to cloud computing is not decentralization - it's centralization, and we're all doing it. Imagine if an attack like this was against AWS... we'd all be screwed.

[traskjd]

Interestingly, in some ways this is a big selling point of AWS/Azure/Goog. The absolute scale they can handle is way up there.

The downside of course, is that whilst their infrastructure can likely handle it, handling the bill associated with 'just scale up your service' could be worse than the attack itself.

[benmac]

AWS has considerable defenses against DDOS attacks of all types - here's the video from Reinvent 2015 which introduces many of Amazon's defenses as well as best practices - https://www.youtube.com/watch?v=Ys0gG1koqJA

Interestingly, the presenter notes that Amazon had seen a drop in DNS as an attack vector in 2015. I asked the presenter (Product Manager) why they hadn't productized the DDoS attack dashboard so you could be aware if you were being attacked (and it was being absorbed by AWS) and his response was that there was insufficient demand at that point to justify the developer staffing. He gave me his card and asked to request the feature so he could us it to make the case internally.

Does anyone here have stories of being successfully DDoS'd on AWS (other than by their own staff :) ?

[stcredzero]

If Azure and Google would like to gain a competitive advantage over AWS, then I would suggest this: Build out a suite of tools for fighting DDOS. Enable private consultants and companies to provide this as a service. Do this in such a way, that cloud customers save money and have to worry about less. Hell, let companies jump in structured as insurance companies! Also bring in cooperation with law enforcement and use data gathering to catch and prosecute DDOS-ers.

[atmosx]

> Enable private consultants and companies to provide this as a service.

If I am an AWS customer I expect AWS to handle/prevent DDoS, same way as they do with S3 to achieve 11 9's availability (the files are saved in multiple AZs in the same region - Glacier IIRC copy files on different regions to avoid data loss in case of physical disaster).

One of the reason for choosing AWS is because AMZ has deep pockets and has the means (financial and technical) to fight against large DDoS attacks, while a smaller provider might not have to do that. Putting clients in a position to have to buy that sort of protection doesn't sound very smart to me.

[takeda]

> do with S3 to achieve 11 9's availability

I see so many people confused about this. Eleven nines is their durability guarantee, their availability that they guarantee is only 99.99%

https://aws.amazon.com/s3/faqs/

[JumpCrisscross]

What's the difference between durability and availability?

[eloff]

Availability is the % of times you try to access your data that you get it back. So 52.5 minutes of downtime a year is still within SLA.

Durability is the % of your data that doesn't die. Eleven 9s means that if you store 1TB on AWS S3 you can expect to lose 10 bytes and still be within SLA.

[gohrt]

durability means you'll get your bits eventually.

availability means you'll get your bits immediately.

[bogomipz]

Durable means it was persisted to disk. Availability means the service is up and reachable.

[bogomipz]

I hear this misunderstanding a lot as well, generally in relation to AWS S3 SLAs. 11 9's of "uptime" would mean service could be be down for 3 milliseconds a year. 4 9s is very respectable.

[ryanlol]

> If I am an AWS customer I expect AWS to handle/prevent DDoS, same way as they do with S3 to achieve 11 9's availability

If you are an AWS customer you should have done your due diligence and know that amazon won't do a very good job at that.

[toomuchtodo]

I don't understand how people who use AWS have such unrealistic expectations.

Someone will always have the upper hand in an arms race, and it's not service providers yet. It's just a matter of finding the choke point between their transit and your code.

[ryanlol]

>I don't understand how people who use AWS have such unrealistic expectations.

Well, the whole point of AWS is not having to deal with the usual hosting stuff. They'll naturally have lots of customers with high expectations and very little understanding of how things work in the background.

[drieddust]

When you are DDOSed they will keep supplying the resources for you to consume and pay them extra. Cloud is commodity so don't expect to be treated like a special snowflake. Your distress is their opportunity to make extra money.

Offtopic but relevant. One of my customer moved their email to O365 without understanding the differences from being ON-Prem. Now they are struggling to adopt their business processes to then limitations MS imposes.

[ryanlol]

>When you are DDOSed they will keep supplying the resources for you to consume and pay them extra.

If the attack is tiny, sure. Otherwise they'll just cut you off.

[drieddust]

> If the attack is tiny, sure. Otherwise they'll just cut you off.

Yet they get to claim inexhaustible capacity.

[qaq]

"achieve 11 9's availability" is this sarcasm?

[_adamb]

They're referring to AWS S3's claim of 99.999999999% durability. AWS actually offers 99.99% availability.

[qaq]

Oh that at least could theoretically be feasible considering AWS's SLA though they might as well claim it's 99.9999999999999999999999999%

[wcummings]

It all depends how you measure it

[Taek]

I thought you had to pay substantially extra to get files stored in multiple regions.

[novaleaf]

i'm not an AWS customer, but from what I have heard, you would be financially responsible for the DDoS traffic bill.

Amazon might wave the fee, but you are the first party responsible.

[capecodcarl]

Doesn't Google already have the infrastructure to deal with an attack of this magnitude? I remember recently reading about Krebs on Security moving to Google's Project Shield service: http://www.zdnet.com/article/google-rescues-krebs-on-securit...

[nstj]

afaik Shield is for select journalists only, not for typical web infrastructure.

[jtl999]

https://support.google.com/projectshield/answer/6358116?hl=e...

> My website is on Blogger, Google Sites, or Google App Engine. Am I eligible?

    > As Google products, these sites already have similar DDoS protection to Project Shield. Your website would not need to be set up with Project Shield. 

Wonder if that answer includes Compute Engine. Doubt it.

[brazzledazzle]

It would be interesting to try using App Engine to simply proxy traffic. I don't know enough about it to even know if it's technically feasible. I imagine the downsides would be many but it could be useful as a temporary measure while you're getting attacked.

[bogomipz]

You can't just build a "suite of tools" and give them to a customer to fight a DDOS. The way DDOS is mitigated is by making routing changes at the network edge. This is not something you want a customer to be able to do for obvious reasons. And these in themselves are sometimes not enough and DDOS mitigation will require coordinating with transit providers, again not something you would want put in a customer console.

[matt4077]

Cloudflare is targeting that market pretty heavily.

[JshWright]

Yup until this morning, AWS was using Dyn as the sole provider of nameservers for the us-east-1 zone. So this attack did have a pretty substantial impact on some AWS services until they updated us-east-1 to use the more diverse set of nameservers their other datacenters use.

[deegles]

That's a good point. If anything, it makes DDOS attacks more effective since you can't easily scale up your bank account :)

[unexistance]

so it will be eventually Cloud VS DDoS eh, both can scale indefinitely so the limit is money, which makes the DDoS guys wins, they practically stole CPU/RAM/NET where cloud providers need to buy hardware as usual

Unless we can somehow secure every net-connected devices, ha (I don't know whether to cry or laugh right now)

[losteverything]

Can you third grade down your comment please?

I find your language to be of high interest like you had a "dUH" moment - which I am ignorant to get myself.

The Sons rays meat

[cheepin]

DDOS usually occurs via a botnet of infected networked devices. Thus, the attacker is getting their resources for "free" since their host is unknowingly wasting CPU and bandwidth during the attack, while the defender is paying for theirs.

[losteverything]

Thanks

Is this analogy accurate?

I have one road to get home. It got blocked so I create 2 more roads.

I now have 3 Roads to get home. All 3 become blocked. So now I have to make another road.

More roads is redundancy and requires capital.

The roads become unblocked but I now must expect future road blocks.

[kpil]

Until Google starts serving pages from our phones...

[atmosx]

Isn't that the meaning the having multiple AWS regions? :-)

If you want HA at local level you'd go with AWS AZs but if you need real HA you need can do the same at region-level.

Of course not everyone has the money/need to go down that route, but it's possible and even advised for some AWS services.

[Symbiote]

You're correct, it's centralisation, at least for the whole community.

It decentralises that one company's DNS -- instead of having one or two DNS servers, perhaps at two sites, they now have 20, at 20 sites. If someone wants to target them, they're probably better protected.

But it's the same 20 servers as a million other companies, so the chance of those servers being a target is much greater.

[gamegod]

> But it's the same 20 servers as a million other companies, so the chance of those servers being a target is much greater.

Yeah, that's what I was getting at. I feel like my chances of being collateral damage on an attack against someone else is way higher in the cloud.

Even today with GitHub and other SaaS platforms going down, we were all affected.

[smegger001]

The cloud can be more decentralized but it more expensive, Done properly having redundancy across multiple clouds aws, rackspace, google, azure, in geographically different areas with different internet service providers it can be done in a very distributed decentralized fashion, just no one actaully does that. Instead they throw everything on one provider and pray its is backed up and secured by that cloud provider better than the IT guy down the hall they just laid off.

[paulddraper]

If you're talking rendering some VPS's, sure that's possible.

But that's a fraction of the cloud. It's hard to integrate every service the hopeful equivalent of every other service.

[cookiecaper]

This is one of the many reasons AWS and cloud computing in general are way overrated.

I know of a company that pays an AWS bill sufficient to buy the equivalent of their pre-cloud datacenter's hardware every 1.5 months. The extra staff required to perform hardware maintenance would also cost about 2 months' worth of AWS each year (that means they're paying ~3x more than they would with hardware). Yet they moved to the cloud because it's the hip thing to do.

Cloud has upsides and things that are useful, especially for smaller proprietors who can take advantage of cheap droplets from DigitalOcean et al, but for grown-up companies, moving off your hardware shouldn't be automatic.

[brazzledazzle]

I think in some cases it might simply be the means to dump 1) people/groups that just don't have a large scale mindset and 2) bypass business processes that are absolutely not designed for large scale systems.

In that scenario you have a bunch of entrenched groups fighting about capex, capacity planning and budget all to get barely enough hardware to account for what you're doing in the next 3-12 months. Instead of taking a step back and creating a long term simple process for regular growth and replacement they get caught in the weeds because they have very old school mindsets.

Then you have your old school finance groups who are using terrifyingly delicate and complex interconnected spreadsheets to manage hardware expenditures and depreciation while maintaining old school draconian policies concerning CapEx budgets but allowing you to basically go nuts with OpEx.

You could try to change the culture in these entrenched groups who will view your attempts to make things better as political moves against them or you could just say "we're moving everything into the cloud" and make a complete end run around all of the people and baggage. The former is probably the "right" thing to do but the latter is going to let you focus on your product letting you get you back to being competitive.

[duaneb]

It's orthogonal to centralization. Abstracting your infrastructure allows you to easily replicate infrastructure providing the same services.

This is only ironic if you expected moving to the cloud to be what provides the redundancy.

[dredmorbius]

There's also the difference between cabled systems, in which multiple elements can independently support load, and chained systems, in which any given link can fail.

The BBC was affected by the Dyn outage not because they themselves relied on Dyn, but because components of their site did.

[mancerayder]

AWS was affected at one point.

I fully agree with you about the paradox of how, in the intent to de-centralize we centralize into cloud VPSes and managed services.

The real reason for the move is that same showtune that we keep hearing in our heads and wish we could tune it out: it's cheaper to move from physical infrastructure to the cloud. It's cheaper to skimp on security by not updating IoT devices. It's cheaper to skimp on security because features need to come first. It's cheaper to outsource operational management to parties with less expertise in places that pay less. To spend less time securing infrastructure perimeters because it costs money.

We feel almost as if we feel comfort hiding behind heavyweights like Google and Amazon will protect us from the bad elements of the world, where we hear about major breaches every few weeks (eg., Yahoo being the most recent). Will this strategy pan out long-term?

With this DDOS, articles about machine learning picking up better password-cracking/guessing algorithms by having previously analyzed large volumes of passwords, major breaches in the financial world, talk of state-sponsored attacks (a la DNC emails) it certainly FEELS like the Internet has gotten a little bit more wild.

[abluecloud]

AWS was hit today, we saw a spike failures. Got hold of one of AWS guys and they basically noticed that the issue they saw in the US earlier in the day happened again in EU west. Funnily enough they probably could have avoided it if they'd deployed their mitigation to the other zones.

[matt4077]

I'm pretty sure DDos against http resources have become quite hard to pull of, which is why there was string of attempts to blackmail smaller email provider but nothing like it happens to similar startups relying on the web. Even the Linode attacks are only possible because they're highly target at a few critical systems there.

[intrasight]

It's harder, but you can distribute your web resources across multiple cloud providers

[gamegod]

If GitHub and Twitter are struggling with this, what chance do the rest of us have?

[bigiain]

Well one upside of not being a Unicorn is that doubling the infrastructure/hosting costs for a project that's at a "cup of coffee a day" or a "diner and a movie a month" budget isn't a showstopper. Doubling Twitter's infrastructure costs would not be good...