[jerf]

For a long time, I've wondered what would finally be the Securitypocalypse, the thing that finally caused our industry as a whole to take security seriously. These IoT DDoS attacks are as good a candidate as any I've seen in a long time. They are fundamentally very difficult to fix in light of the non-updateability of many of these devices, and this is only the beginning, because the IoT has hardly begun to develop. And in the short-term, I'm not sure I see any hope, because the forces that make people throw out cheap devices with broken firmwares with no update capability aren't going away.

If we could somehow mandate that these devices were supported with firmware updates for the indefinite future, that would simply destroy the entire market. And you can't do that, because even the devices created by an entity that no longer exists and didn't sell its IP to anybody else will eventually be enough to do these DDoSes, if they aren't already.

[gtrubetskoy]

These attacks are mostly possible because of the complacency of operators at many sites and companies. This is not a new problem and many of RFC's talk about methods for preventing and mitigating them, but most people don't care and prefer to just outsource everything to a single provider, which becomes the weakest link.

The Internet wasn't envisioned with a single email provider, single DNS provider, single app container provider. (Ok, for most of these you have two, sometimes three choices, but still, that is too few). The centralization makes everything very vulnerable - imagine what would happen when Gmail is knocked out for a day.

[brongondwana]

Seriously? It's OK if only one site/company gets taken offline at a time?

There's no RFC that talks about methods for preventing or mitigating hundreds of thousands of machines all sending arbitrary traffic at you at the same time.

The only way to protect yourself from that sort of attack is to buy filtering from someone who has a bigger pipe than the largest DDoS available, and have them filter the packets so that you only get clean traffic. Unless you know of an alternative that nobody else has heard of yet.

So you wind up buying transit / scrubbing from one of a few big providers, because that's the only way to avoid being sniped by DDoSers.

[gtrubetskoy]

> There's no RFC that talks about methods for preventing or mitigating hundreds of thousands of machines all sending arbitrary traffic at you at the same time.

The RFCs generally say that the problem is "you", i.e. the target. Of course those device makers could make their devices a little more secure, can't argue with that, it's another form of complacency. Still - the attackers are only able to do this because their targets are few.

If there were thousands of DNS providers such as Dyn each serving a small number of clients spread all over the world, it'd be impossible to attack them all.

To cause maximum damage you need to identify hosts that are common across many big companies. Someone did their homework and figured out that lots of companies are using Dyn for DNS, and for the East Coast of the US this is just a handful of servers. If the same DNS services were spread across 1000 servers, then the attackers would need proportionally more "power" to knock them out. DDos-ing 10 boxes is _so_ much easier than 1000 (approximately 100 times easier, to be precise).

[supergeek133]

The problem with these devices in particular is the weak point is the user. As is the case in most attacks.

Your average user says "Sure I can setup cameras" then sees "remote access" in the menu, sets it up, maybe it has some UPNP to the router and BOOM. Magic remote login without any type of mitigation.

[eric_h]

Indeed. My mom got an internet connected "security camera" kit (for cheap from one of the big wholesalers, can't remember the manufacturer) and asked me to set it up.

The hardware was nice, cameras did a reliable 1080p full color, but the whole reason my mom wanted it was so she could check in while she and my dad were traveling (and also sneak a peek at her bird feeders while she was away; avid birder, that one).

So, I hooked that thing up to the network and did a port scan on it... First noticed - it's listening to port 22, auth is a googleable default password. It supports UPnP to punch a hole through the NAT and serve up video on another port. OS on the server box is some slightly customized version of linux with an _old_ kernel.

So I said, "Sure mom, I can set this up for you. We're going to need to get you a new firewall, it'll probably be easiest to put a *nix box in front of your wifi access point, then we can set up a tunnel between the isolated camera server and a locked down outside server that only you have access to so we can be sure that no one else is looking at those cameras. Should only take me a few hours, and we'll need to buy a box to run the firewall, and then a small monthly fee to keep the internet accessible server running"

Her response, "but it says on the box that it's easy to setup for outside access!". Mine: "It's easy to setup for everyone to access, much more involved if you want to make sure it's only you who has access".

Admittedly, it did have some authentication for accessing the video streams, but I didn't trust that thing as far as I could throw it; I'm glad she decided not to go through the trouble of getting it working (but mostly because I'm lazy and didn't want to have to setup and support that damn thing).

I can only imagine that the people who bought that device and didn't have a security paranoid person to help them set it up are all contributing to this most recent DDoS attack.

[merb]

    Her response, "but it says on the box that it's easy to     
    setup for outside access!". Mine: "It's easy to setup for 
    everyone to access, much more involved if you want to 
    make sure it's only you who has access"

well that was a pretty clever answer, I needed to laught about that :D Basically the commercial was right :D "easy to setup for outside access" that didn' implied a single person ^^

[smsm42]

The problem here is there's nearly zero incentive to do it right. I mean, ok, let's say the worst - somebody breaks in the box. For a regular person, worst thing somebody would get access to their DVR. As long as it keeps working as DVR, they couldn't care less. Yes, this DVR would also serve as botnet bot, but the owner doesn't care. It doesn't hurt them - except when Twitter goes down but they don't make the link between them not configuring the DVR properly and Twitter going down. Until we find a way to make the incentives work in right direction, nothing really would change...

[paulddraper]

> Admittedly, it did have some authentication for accessing the video streams, but I didn't trust that thing as far as I could throw it

So...you wanted to have authentication and it has authentication...I must be missing something.

[stephen_g]

It may not have been over HTTP, so possible to be sniffed. Or, even if it did have HTTPS, it might not generate keys in a secure way (or might use the same certificate as other devices). And you don't know if there are hidden backdoor accounts that might be found eventually...

So, yeah, it makes sense to block it - personally I block IOT devices from the Internet entirely (and don't let them initiate requests to my local network even) and use a VPN (IPSEC/IKEv2). That wouldn't work for devices that connect to cloud services, so I'd have to set up new firewall rules if I got one of them.

[eric_h]

Late response, but yes - there was no https support whatsoever on this thing. Authentication was some custom shit and intended to be passed over the internet in clear text.

[gaius]

So...you wanted to have authentication and it has authentication...I must be missing something

You missed that you could SSH into it with a default password that is easy to find on a web search.

[paulddraper]

So... don't use that default password?

[drieddust]

I will be interested if you take time to write this up.

[smn1234]

why not generate a cert based off mac address and allow customer to use that

[redleggedfrog]

The real problem here, and this isn't going to be a popular position, is that you're relying on the internet for important things.

The original engineering and architecture of the the internet (and the web) was not intended to create something you put all your eggs in. It was for sharing information, not building your mission critical business operations on.

Right now, if you dumped your business into a cloud service you're mostly dead in the water. But those who have local infrastructure can keep working. As people have been noting here, centralization is bad.

[fzzzy]

Actually, the original engineering and architecture of the internet was intended to provide reliable command & control in the event of a nuclear war. A network of last resort. I can't think of anything more mission critical than that.

[late2part]

No, it wasn't. That's a myth, disturbed in many sources, including [1]. Also in [2]:

Many people have heard that the Internet began with some military computers in the Pentagon called Arpanet in 1969. The theory goes on to suggest that the network was designed to survive a nuclear attack. However, whichever definition of what the Internet is we use, neither the Pentagon nor 1969 hold up as the time and place the Internet was invented. A project which began in the Pentagon that year, called Arpanet, gave birth to the Internet protocols sometime later (during the 1970's), but 1969 was not the Internet's beginnings. Surviving a nuclear attack was not Arpanet's motivation, nor was building a global communications network.

Bob Taylor, the Pentagon official who was in charge of the Pentagon's Advanced Research Projects Agency Network (or Arpanet) program, insists that the purpose was not military, but scientific. The nuclear attack theory was never part of the design. Nor was an Internet in the sense we know it part of the Pentagon's 1969 thinking. Larry Roberts, who was employed by Bob Taylor to build the Arpanet network, states that Arpanet was never intended to link people or be a communications and information facility.

[1] https://www.amazon.com/Where-Wizards-Stay-Up-Late/dp/0684832...

[2] http://www.nethistory.info/History%20of%20the%20Internet/beg...

[fapjacks]

Where Wizards Stay Up Late is a fairly dry book, but it contains interesting kernels of information (like this). It's not a page turner, but it's worth a read if you are interested in things like, for example, the information in this comment's parent.

[eternalban]

A few oral history interviews with key actors also confirm this.

[mycall]

> Arpanet was about time-sharing. Time sharing tried to make it possible for research institutions to use the processing power of other institutions computers when they had large calculations to do that required more power, or when someone else's facility might do the job better.

Arpanet is distributed shared information for science. Nuclear technology is science. Surviving science is a war that requires nuclear insights. Therefore, the Arpanet was developed for surviving nuclear war.

[braveo]

And yet I suspect the government could pick up a phone, hop in a vehicle, etc and communicate with the right people.

Which means it falls under what he said.

[cognivore]

ARPANET is nothing like the monstrosity we have today.

[supergeek133]

Even still, if indeed it is the cameras doing this, it's a problem of our own creation. The internet 'is fine' without a botnet of dvrs.

[cmdrfred]

Exactly, I have tons of IOT devices. I put them on a separate subnet that does not have a gateway to the internet then I VPN into that network to access them. Perhaps a product that makes that a simple process will solve the problem?

[NetStrikeForce]

We partly do that at Wormhole. I say partly because you still have to be able to access one of our addresses. Port of last resort is 443/TCP, so it works on lots of tricky networks out there.

The idea is that all your IOT stuff establishes a connection to this server, creating an encrypted network between them. You then add your control servers to that network and job done. You devices don't need any inbound access to talk to each other. All the connections are outbound, so no ports to open on your firewall and no risk.

You could do this by yourself, but we take that hassle out of your hands. Happy to help with custom deployments too outside our main service; it's a great way of learning our customers' needs.

It's hard though to have your exact setup as a service, it implies incoming VPN connections to the site where you deploy your IOT and a VPN server of sorts.

Our main focus was remote teams and devs having to use remote servers, however IOT might be a killer use here.

https://wormhole.network

[cmdrfred]

Interesting, I have a few thoughts. Perhaps you could sell a preconfigured pfsense box (or make a raspberry pi image to start with) that when plugged into the customers router creates a reverse tunnel via your service as well as a WiFi hotspot. Then offer the user a very simple firewall control panel and they can choose what devices to allow to the open internet and what to keep private and accessible via some sort of authenticated channel. Thus devices that contain sensitive data or require enhanced security (cameras, private network attached storage devices, home automation) and devices that require internet access (Amazon Echo) can both be served by the service.

Very nice service by the way. I have used ngrok in the past and found it invaluable for a few odd applications. I'll give it a try in future.

[NetStrikeForce]

Hi!

Thank you for the feedback and the suggestion. It is a good idea actually. I'm considering new features in the roadmap, because at the moment I don't even offer Internet access through my system, it's just a private LAN (I'm not competing with the myriad of privacy-minded browsing VPNs out there). Adding a manageable Internet Gateway could be a nice option.

Developing and deploying a software+hardware piece would be very interesting too, so there's no need to deploy agents on the remote servers or IOT devices (on most of them you probably can't) and I take the hassle out of my customer's hands to setup a e.g. Linux gateway to route traffic through the tunnel.

[RickS]

FWIW, I would definitely be interested in paying for a service like this. I'm technical enough to care about this, but not technical enough to solve it myself. Similar to where I was before dropbox.

[NetStrikeForce]

My comment here might be relevant to your interests: https://news.ycombinator.com/item?id=12765051

It could suit your needs or we can help with custom deployments. In any case I'd like to learn more about your needs and your expectations. Can I drop you an email?

[RickS]

totally, it's in my profile

[voltagex_]

I've been thinking about how you'd design a UI for that, that was easy to use. Maybe a separate wifi network that IOT devices go on to, and then a web app that knows devices with XYZ MAC are LIFX bulbs and shouldn't be able to talk to the smart TV, but that phones on the network should be able to jump the subnet and talk to the bulbs.

[cmdrfred]

You can make it semi-automated in a way. I believe the first 6 characters of the mac address are the vendor id, I'd get the DHCP server to assign different vendors into different isolated vlans but with short leases at first and then allow you to merge them, assign permissions and move them around. Call it "learning mode". It won't be perfect but you can also augment it with human created presets.

[mannykannot]

The problem with any solution is getting it used widely enough to make a difference. We seem to have an unlimited predilection for making the same mistakes repeatedly, even though we could avoid them.

[patcon]

heh. move over user-centered design, user-centered malign is making a come-back :)

[NKCSS]

It's easy to fix; back in the day when a machine was infected; an ISP would just block outgoing traffic, contact line owner and re-enable when the issue is resolved.

[gaius]

If the "machine" in question is my ADSL router as supplied by my ISP, I will be deeply unimpressed if they block me due to their own negligence in updating it!

[Jtsummers]

Similarly, a single bad device on my network would block the whole of my network from the internet. It's another sort of denial of service attack.

We need IPv6 and have devices either access the internet with their own IP address or not access it at all. This solution, then, would only impact bad actor devices, not your other (non-compromised) devices. Still, not easy.

[cpncrunch]

I think it's fair to block the entire network. It is then up to the network administrator to fix the problematic device.

[Jtsummers]

While technically accurate to describe them as such, the vast majority of consumers (and internet service subscribers) lack the actual technical expertise to be network administrators.

Where these devices are being attacked inside, ostensibly, professional organizations (companies, schools, government buildings), I agree. But there you have, again ostensibly, an actual network administrator capable of dealing with the issue (and paid to do so).

[matthewmacleod]

I think that's okay.

We don't expect all homeowners to be, say, experts in electrical wiring, or gas supply, plumbing, drainage, or waste management. But all of these things—if they are poorly modified, managed, or maintained—can cause impacts on third parties. In the case of networked devices, the possible impact on third parties is even greater. We also enforce strong regulation on these systems – defining what may and may not be legally connected to public utility networks, for example.

We would probably expect a homeowner to hire a tradesperson to maintain these services, and in some cases it's legally mandated that only a qualified person may install or modify these systems. Is it then unreasonable to kick consumers off of the Internet when they install poorly-maintained devices, and require them to resolve the problem – perhaps by hiring the networking equivalent of a qualified plumber?

[guelo]

That's fine, if you connect some cheap webcam and it causes you to be knocked off the internet you're going to be mad, leave a bad review for the camera, and not buy from them again. Market forces would then incentivize better security to be built into these devices.

[notatoad]

"the vast majority of consumers (and internet service subscribers) lack the actual technical expertise to be network administrators."

that's true, but the vast majority of internet service subscribers aren't their own network administrators. If you're using an ISP-supplied modem/router combo, i'd say that your ISP is your network administrator. If my ISP wants that kind of access into my local network (and they don't give me any other option) then they should be doing some actual administration.

[rev_bird]

"Fix" is a relative term, especially if IoT devices are in play – yes, turning off the internet to customers stops the attack, but then (at least?) thousands of people lose internet connectivity because of a vulnerability that they could very well be powerless to fix. I'm not saying it's ok with me that an army of smart refrigerators could be taking out big chunks of the web, but it's a lot easier to tell someone, "Hey, either get the infection off your computer or re-format" than it is to make someone buy new lightbulbs and appliances.

[zzleeper]

Not powerless, just unplug their toaster and they get their internet back.

What is powerless is that many people today couldn't get twitter, github, reddit, spotify, box, etc. because many people don't care about securing their webcam.

[eridius]

I would hope things like smart refrigerators and lightbulbs actually still operate normally when the internet is out, right? By "normally" I mean similar to "dumb" versions of the same product. So a customer could fix the issue by kicking the device off the network (disconnect the smart fridge from the ethernet / wifi, unplug the hub for your light bulbs, etc) without actually having to immediately replace them.

[rasz_pl]

When a pipe breaks in your condo and starts flooding all the people below nobody asks which appliance might be leaking. Water is cut and you get the bill for _all of the damages_.

[sparky_z]

Possibly stupid question: why is that no longer done?

[JoshTriplett]

Because it's hard to get an ISP to disable a service for one of their paying customers to help other people on the Internet who aren't paying them.

[zzleeper]

Why can't everyone else then block the customer? Get the big 5 tech companies to block IPs that are shown to do DDOS, for say a 24hr period, and you will see how quickly they unplug that IOT Toaster

[Jtsummers]

Speaking as not-me, the average, non-technical homeowner who just installed his new internet connected washing machine at home.

Great, now I can throw in a load and get a notice on my phone when it's done. This is awesome! (3 hours later) Wait, why can't I get to the internet? I call my ISP, they tell me that my connection is fine (it's tech support, they aren't security experts). But, I tell them, Google doesn't work for me. They do some tests, everything should work. I bitch, moan, cry a little, rage quite my ISP and sign up with someone new. It works for a few days until my washing machine (having been offline for a bit) gets exploited again.

I still don't have a clue as to why I'm being blocked from Google and company. Maybe they kick back a message as a 4xx (what would be appropriate?) that says my network has been hacked. But I've seen those sorts of things all the time in ads, I know that's just someone trying to scam me, convince me to run something that'll install a virus on my computer.

Must be my computer! Damn Dell piece of shit. I can't afford a new one. Maybe that neighbor kid can come over again and help me out with this.

($200 and several trips for the neighbor kid later it's still not solved)

[empath75]

From my point of view as someone who is no longer ddos'd, I don't have a problem with this.

[zzleeper]

As you said, some sort of message would have to be the way. A 4xx probably won't cut it but something like the messages Google shows you when asking for a captcha is fine.

My point is that there will be a cost, and that taking action against vendors won't be enough (sp. if they are in a different country, are no longer in business, etc.)

[pavel_lishin]

> Maybe they kick back a message as a 4xx (what would be appropriate?) that says my network has been hacked

429 seems appropriate.

Or maybe even 451.

[cortesoft]

Not very quickly? First, you wouldn't know why you were disconnected. You would try the standard things first (plug and unplug your router, etc). Then maybe after a while you would call your ISP. Get put on hold a bunch. Your ISP tech support probably won't know much either, since in your scenario it isn't the ISP doing the blocking. They MIGHT test the connection, or maybe they just give the customer a new IP address.

It is going to take quite a while in this scenario for the user to realize it is their IoT toaster that is causing the issue.

[dom0]

Because today you can't call the customer anymore if you block their traffic.

[rasz_pl]

and dont forget the part where user is charged $ for violating contract.

[seanp2k2]

Yep, and manufacturers have not much incentive to update firmware for a device which is not their latest greatest or update firmware while not adding more features to help them sell more. Security isn't a feature that the vast majority of consumers would pay extra for or know how to verify anyway. There was plenty of demand for that one "unhackable" android phone, but I'd be blown away if it wasn't 100% snake oil.

My prediction is that it'll get worse before it gets better and that these type of botnets will be around for at least 5 years. Look at what happened to unsecured-by-default routers, android phones, Windows PCs, cars...the way consumers will get more secure stuff is by manufacturers being publicly embarrassed / sued over problems until caring about security makes business sense, then they'll have it in their hands when their old insecure gadgets die.

My cynical side side thinks this will be a problem until all the old endpoints supporting these insecure things are shut down eventually in 5-10 years.

[Angostura]

This isn't just small manufacturers either. I bought a new Samsung tablet for my kid two weeks ago. It is running a three year old version of Android with no updates available. Pretty shocking.

[JoelBennett]

There'd be something ironic about a manufacturer's website being made unavailable because of a DDoS caused by their own poorly secured devices.

[trhway]

>They are fundamentally very difficult to fix in light of the non-updateability of many of these devices

as you proved, fixing the situation by fixing the devices wouldn't be a feasible approach. The traffic from those devices is carried by ISPs and this is there this traffic should be stopped. To me the situation reminds about email spam. We didn't get rid of spammers, instead the email traffic is analyzed and dealt with accordingly. I'm sure that ISPs easily see the patterns of such massive DDoS attacks and could just drop (or throttle down into oblivion, like 100s times down) the participating traffic.

[HipHopHacker]

> For a long time, I've wondered what would finally be the Securitypocalypse, the thing that finally caused our industry as a whole to take security seriously.

Nothing. If the economic system revolves around capital's valorization of itself, security is a distraction from that. I have to spend five seconds typing my password in every time I sit at my desk? I can't just easily e-mail this executable file to my co-worker and have them run it? My desktop is locked down by the desktop admins to prevent me being able to do this, and many other things? Every implementation of security costs money for the personnel to do it and possibly the product cost. Plus any lost productivity it might cause (15 seconds to type in a password each time one sits at their desk, compounded).

Donn Parker wrote one of the first books on computer security in 1976, Crime by Computer. The opening words are as apt for corporate security now as it was then. The #1 fear for the corporate manager are the employees of that company. They are the ones with the greatest control over the means of production, so to speak, even more than the managers themselves who are de jure in charge, but are de facto one step away from actual control. Look at how much access someone like Snowden had at Booz Allen.

Obviously, if all products have wide open holes, script kiddies will be able to get control. Some minimal security will always be done to stop this sort of thing. On the other hand, one (or better yet, several) dedicated people who want to get past some security arrangement can almost always get in. Even if the firewall is supposedly impenetrable, the wifi or the building security or the social engineering credulity of employees or something will be there. There will be some weak link in the chain. Especially for a company that needs to make a profit.

The real security is that semi-intelligent, persistent agents that seek to access and control systems without authorization are lacking. Things depend on the conditions that cause this to rise or diminish. Because once it rises, there is little that can be done. I forget who said that the czar's Russian Okhrana was one of the largest, most extensive security forces that existed. That meant little when Russia began collapsing in 1916 though - all it meant was that they were even more aware that virtually everyone in the country was becoming the czar's enemy.

Securitypocalypse events due result in business and government putting more focus on security for a while, but time moves on, and attention drifts back to the main focus. These things go in waves, and total security is never something of the highest priority.