[tiredofcareer]

Some hopefully-helpful clarifications of the inside baseball talk from just the overview (I haven't read the full zine), enhanced with inside and general knowledge I've gained in my travels on this mortal coil:

- HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes.

- In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though.

- Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators.

- Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April.

- The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons.

- Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it.

- How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access.

So ends clarifications, thus begins conclusions:

- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.

- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.

- Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider.

- EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done.

- EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails.

- SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.

- Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture?

- HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time.

- Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.

[JshWright]

> - Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.

Unfortunately, there's not much slack left to cut. Linode pulled that line taught with the last major breach of their system (and their subsequent opaque response).

It's clear Linode's security practices leave a _lot_ to be desired, and their response to this incident, while better than the last, didn't go anywhere near far enough in terms of transparency (as well as demonstrating some fundamental gaps in their understanding of the current state of infosec).

[tiredofcareer]

Agreed. The one thing I think Linode could do better is communicate. The secrecy model is ... odd.

[freehunter]

I didn't pull my hosting from Linode because they got hacked, I pulled it because they were hiding this from me. I even went as far as replacing the card I used to pay them and dealing with the massive headache of updating payment info with a new card number because I didn't know if I could trust their assertion that CC numbers didn't get released.

If I can't trust you at your word, you no longer have the privilege of holding my data or my CC info. Two breeches with a lack of communication really does spoil overnight the trust that has been built up over years of wonderful and reliable service.

[gingerlime]

I'm curious to know who you moved away to? and what gave you the confidence that they are a) better at security b) better at communicating transparently when things go wrong.

This security incident is very upsetting, but for me the linode track record in communication, responsiveness and support is still pretty amazing compared to the competition. At least at this price range.

AWS might be safer, but I basically don't get any support (for a reasonable price like I pay linode), Rackspace didn't seem nearly as responsive or transparent on much more trivial matters. Who else is out there which is worth switching to?

(I'm not being sarcastic, I'm genuinely curious)

[threeseed]

I have found lots of the smaller VPS providers (especially those that provided dedicated/colo services) to have great service. Check on http://webhostingtalk.com

I fail to understand how you can think Linode has a great track record. It is an unequivocal disgrace. TWICE they have mislead their own customers over a major security incident. And there have been lots of times during outages (in particular my time at Fremont) where they were MIA.

[lelandbatey]

I give Linode a lot of slack. When people say "Oh, they should be more secure" I often say "Really?"

In an ideal world, yes, they should be more secure. However, as in this case, they got taken advantage of via a zero-day attack, with others planned well outside the scope of what Linode could have planned for. Which is insane. Can you even name something, anything that they could have done to protect themselves? Additionally, given the unique form of attack, figuring out what was going wrong was probably not possible. Thus, they knew as little as you did.

And then, everybody switches to some other provider. But do they switch to "super secure, we examine every byte of the software that we run to make sure we're bullet proof" hosting provider? NO, everyone just switches to another commodity VPS provider that is vulnerable to all the same super high level attacks that Linode is vulnerable (maybe even more attacks, given that Linode actually has a tremendous amount of experience).

In reality, you're only getting more security by switching to a less prominent hosting provider, A.K.A. security through obscurity. Which is the worst kind of security because it's not secure at all.

It's like getting mad at the mayor of your city when a meteor falls on your house: unproductive and misguided.

[freehunter]

I moved away to me. I enjoyed having my test server in a data center so I could work on it from anywhere, but it wasn't worth risking a security breech where no one would tell me I had been compromised. Instead, now I somewhat inconveniently host it on a machine sitting in my basement.

I'm merely a hobbyist/researcher, so I don't have a production environment to worry about.

[gingerlime]

I see. For me this is unfortunately not an option.

I need to be able to serve my customers, and they are all over the world. So I need a provider with data centers in multiple global locations, that offers an API, that has decent support that responds quickly, and at a similar price range... Interested to find alternatives that are also more secure and better at communication.

[kijin]

> The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for.

If they had access to the database, it may have been possible to delete malicious jobs from people's histories. Even if the user had email notifications turned on, an attacker with full access to the database could have turned it off temporarily (just flip a boolean flag).

[tiredofcareer]

That's a good point and I hadn't considered it. It still reboots your Linode, though; worth considering a 'echo "I just rebooted! Did you expect that?" | mail' in your rc.local for this reason, since a reboot should be an infrequent event.

[singlow]

yeah, but will likely reboot it into recovery mode so they can remove anything like that from your boot sequence anyway, and if they don't notice it, they will be done by the time it goes out anyway. External monitoring is the best way to notice the node went down.

[soulclap]

I already upvoted this but wanted to take the extra time to say thanks for your post.

This whole thing really makes me wonder how many hacks and deals behind the scenes are going unnoticed.

[blibble]

> For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes.

what's stopping the bad guys from just proxying dns queries they don't care about to the original NS?

with this sort of trickery you could get a "domain control validated" https certificate too!

[tiredofcareer]

Hence why I hedged impossibility, and while I can think of a way that would work, it'd be tricky. You could probably hack BIND to do this (in resolver mode) fairly trivially, but I'll defer to the actual security experts here of which there are many to shed light on whether such an attack is commonly observed in the wild.

My usual suspicion is that in general, the volume of DNS traffic should give you pause before you start putting custom code in the path of answering a query. Clearly it's possible -- Route 53 is built upon that very notion -- and I suppose in this scenario it's feasible.

Don't forget every Linode has a hostname under linode.com. I think splicing yourself in and running a conditional on every query would overwhelm whatever you point the firehose at and you'd have to plan accordingly. All it would take would be to add a couple hundred milliseconds of latency to the average DNS query (even before the inevitable carpetbombing of p99 latency) and a competent high-traffic administrator is going to start looking around.

[blibble]

DNS is very compact (a few hundred bytes a query), you're talking about maybe 10mbit of traffic tops: not a hard problem these days

[tiredofcareer]

It's not about size, it's about rate and introducing latency. Just the hijack itself is going to add DNS latency, which is monitored by any competent operations team. Expert operations teams, and I know of one, also monitor the BGP path to their public addresses (including nameservers) to detect things like the Youtube kerfluffle.

Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it. And you're still going to introduce latency by inserting another hop. That's my point, though I do agree with you.

[riotingpscifis]

>Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it. And you're still going to introduce latency by inserting another hop. That's my point, though I do agree with you.

Welcome to the world of recursive name servers, there is a lot of software out there that does exactly what you just mentioned, I fail to see what would be hard about making this change.

[sillysaurus]

Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it.

Hm, why? Any modern CPU is blazingly fast. Writing it in Ruby probably wouldn't be smart, but Python + PyPI or Lua + LuaJIT would easily get within a factor of 10x of C.

[tiredofcareer]

I didn't say it would be technically impossible, I said it would be noticed. If you make it a theoretical problem, and it most certainly isn't (there are a lot more practicalities involved), you're adding at least another string compare to every query. That's enough of a latency shift for me to notice in my graphs -- I notice when the Internet reroutes itself and my DNS latency goes up by 5 milliseconds.

This isn't a "could it be done?" exercise, it's more of a "could it be done without detection?" exercise. For this specific case, it's a pretty big risk.

[blibble]

hell, you could run it on linode and probably be right next to the box you're proxying... ;)

[MichaelGG]

>with this sort of trickery you could get a "domain control validated" https certificate too!

I don't understand why HSTS didn't allow some sort of pinning, or the ability to specify a certain kind flag in the certificate is required (like EV).

[VLM]

"because you know, that's a good target to burn registrar access on and all"

No, nearly ideal use. Its like a strategic nuclear weapon. You don't use it sneakily, that's the opposite of the whole point. Always intimidate as publicly as possible and in the tech community messing with linode is about as public as it gets.

The other part is showing off, its like declaring "we have access to them all but we don't care about name.com". Maybe they do...

[tiredofcareer]

They didn't utilize the access to go after Linode. They intended to utilize it to go after SwiftIRC, which nobody gives a shit about. That's where my comments came from. Linode just happened to be a nice prize on the way.

[jrochkind1]

Yeah, I mean, it's supposed to come off as showing off, right? "Sure, we're so crazy good, that we can take out name.com and linode just cause some script kiddies pissed us off, no sweat. Don't mess with us. And we're so in it for the lulz, that' SURE we'd take out linode and name.com just to get some script kiddies, and not bother trying to sell the CCs or anything."

Or, they're lying. I mean, they could be lying about the whole damn story, who knows (not me).

[meowface]

You hit the nail on the head right here.

Assuming the account is completely true though, I would say they succeeded in showing off. It's pretty impressive that they took over name.com and discovered a CF 0-day just to get into Linode, all just to fuck with one IRC network no one cares about.

[ak217]

Thanks for the overview.

> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.

Actually, the biggest lesson I'm taking away from this is to never trust any one piece of software, and to always have multiple lines of defense/alerting sitting on independent software stacks.

[marshray]

> The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. ... the access they obtained does not lead to root on the Linode host fleet itself

I wouldn't bet on that.

> There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.

Isn't that the same logic Everydns/Dyn Inc. used when they censored Wikileaks?

> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.

True dat.

[tiredofcareer]

> I wouldn't bet on that.

I don't need to wager, and can speak with authority based on what I know (which I'd prefer to leave vague). There are two vectors into a Linode's filesystem from the perspective of an internal attacker: having root on the Xen host or gaining a login on the Linode. Knocking over the database and Web server gives you neither unless the person reused their account's password as their root password, in which case it's behind a cryptographic hash and subject to the typical rules there. If you own the database, you do have LISH access which gives you the equivalent of a VGA console; if someone left that console logged in, it's a vector as well.

The only vector HTP would have had in the general case would be bouncing the Linode. It's a fairly sufficient air gap, in a way.

[marshray]

Thanks for the clarification.

ISTR being able to back up Linode images, migrate them around, and perform some other admin tasks from the web interface. So I still wonder if the "air gap" API is a bit more powerful than simply the ability to request a reboot. But, again, I don't know.

[tiredofcareer]

Those buttons just instruct the hosts to do things and don't actually have power themselves.

[marshray]

The buttons belie the existence of an API that can "instruct the hosts to do things". Some of those "things" are pretty powerful.

Without knowing what those things are, I'll just take your word for it that none of them could ever possibly be leveraged to compromise a host or guest without unmaskable and permanent messages appearing in the logs.

[eridius]

You have to decide whether to take HTP at their word that they deleted credit cards.

Don't we also have to take them at their word that they even had decrypted CC's? I haven't seen any proof yet that they obtained the decrypted private key, just their statement saying they grabbed in-memory keys.

[marshray]

If they pwned the server that accepted the HTTP(S) POST with the payment information, they were in a position to obtain at least some CC numbers. They were probably also in a position to obtain the keys by which the CCs were encrypted.

[eridius]

The encryption keys are useless. The decryption keys are what's important. You do have a point that they could have theoretically intercepted HTTP(S) POSTs, but I don't think anyone's claimed that they actually did that.

[marshray]

They were using asymmetric crypto on their CC data column?

[eridius]

Yes. Linode publicly stated that they were using public-key cryptography, that the private key was secured with some crazy-long passphrase, and that the passphrase wasn't stored digitally, meaning that once a month when they billed they had to manually enter the private key.

So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely), or the hackers must have had control of the machine during the time in which they did billing (which I don't think is true, because billing presumably happens either at the start or the end of the month, and didn't the hack take place a bit earlier than that?).

So yeah, I believe them when they said they got the private key. But nobody's said anything to convince me that they got the _decrypted_ private key. And if the passphrase really is as long and complex as Linode claims, then it should be reasonably secure (caveat: I am not a security researcher, or otherwise qualified to judge the security of anything).

[tiredofcareer]

> So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely),

They bill you the moment you add a Linode, automatically, if your credit is not sufficient to cover the new Linode. Careful walking that assumption too far; I think it's safe to say the key was kept in memory.

[marshray]

OK, I remember reading that now.

Good move on their part.

[nikcub]

re: your first paragraph, there is a way to jack an entire DNS record and not miss anything. It involves writing a custom DNS server. Once you become the primary, as the queries come in, if you are queried for a record that you don't know, you simply forward the query to the old primary and then store it yourself. Works all the time.

Domains are jacked and proxied a lot more than people know. The hackers have custom tools (rather than Squid + BIND etc.) that perform these tasks and keep them hidden.

Even better, you could host spoofed DNS for a number of Linode's on a single small 128-256MB virtual machine. The infrastructure required is tiny. Definitely possible, definitely happens all the time.

[white_devil]

It bugs me that apparently, everything I'll ever host can just be "owned" at will by some random bunch of hackers doing whatever they feel like doing. Is it feasible for a "mere mortal" to stay safe?

[cparedes]

Security is about mitigating risk, not about eliminating it.

Keep up with CVE's, don't provide a wide attack area (so lock down interfaces to your machine and don't expose much to the world), and keep blast radii as small as possible (so even if your machine does get owned, you can possibly restrict it so it doesn't automatically mean they gain access to other systems in your network.)

Oh, and model the threats to your network/application. Make sure you're securing against the right threat. As an example, anti-malware is wholly ineffective against social engineering - maybe it's more productive to train employees and make sure that each employee doesn't have total access to all privileged systems.

[tiredofcareer]

User input is trying to kill you until proven otherwise. And even then, it probably still is. If you accept anything from a user, treat it like you're carrying spent nuclear fuel around your application; that doesn't just mean form inputs, either; sometimes you have to wear gloves that even consider files read from the filesystem as suspicious. There are numerous attacks on temporary files if implemented incorrectly.

Damned near every exploitable security problem in an application can be traced back to this one simple rule. XSS is a common one. The ones that don't conform to the rule are rare and magical, like unicorns, and usually involve something exotic like hardware side channels. If you take input it will be abused, so plan accordingly. That occasionally manifests in unexpected ways like timing attacks, in which case an attacker repeatedly sends you lots of carefully-chosen input to deduce something based on how long it takes your code to reply, much like cracking a safe. It's a basic attack on a string compare, which is O(n). This simple code is vulnerable:

    def price_multiplier(form):
        if form.input == 'super-sekrit-coupon-code':
            print >>browser, "100% discount applied!"
            return Decimal('0.0')
        print >>browser, "Sorry, that's an invalid coupon!"
        return Decimal('1.0')

A timing attack would try to deduce each letter here, since the string comparison will short-circuit once it finds a wrong character. Using a simplistic model, say I sent 'a' through 'z'; 's' took 10 time units to respond, but the other characters took 8. Now I try 'sa' through 'sz', and go from there. As a thought exercise, try to imagine a few ways to prevent it, and consider the pros and cons of each; now you're thinking in the security mindset.

Securing an infrastructure is another story, but the rule is applicable with the occasional clarification to everything there, too. There are domains of trust even within your own organization; malicious employees will try to harm you, as well, and you get little to no warning of those. Do your interns need root on machines containing customer data? Employees are users too.

[white_devil]

Well, it's somewhat comforting to know it's mostly about user input. Thanks! Not that it's easy to secure input handling either.

I've seen that kind of timing attack discussed somewhere, and the solution there was to do some kind of byte-by-byte comparison that would always take the same amount of time. It makes sense.

[caf]

If you've redelegated a domain which you don't have a full copy of to your own servers, couldn't you just proxy all the requests that you don't want to hijack to the original name servers?

[tiredofcareer]

To save vertical space, I'll refer you here: https://news.ycombinator.com/item?id=5667471

[tomjen3]

I don't give a shit about my (former) linode servers and never want to have anything to do with the bastards again.

I just want to know one thing: did or didn't they leak the credit cards?

[zoul]

> I just want to know one thing: did or didn't they leak the credit cards?

Does it really matter? If your card was used with Linode, it should have been blocked by now anyway.

[tomjen3]

Yes, because it is a huge pain to go through for no reason.

[mcintyre1994]

Not publicly, as far as I know. They claim to have deleted them, but I don't think it's exactly possible to prove you've deleted every copy of digital data. You should probably assume they're compromised.

[ikawe]

Allegedly no, the deleted the breached data on the condition that Linode mention them in the security release (from the article).