[chubot]

So what defenses should an organization employ to prevent these types of attacks?

From this non-technical article, it looks like they penetrated employees' computers and used their credentials, which makes sense because it's probably the weakest link.

It reminds me the philosophy/motivation behind Qubes OS [1]: there is no server security without client security.

What are banks running on employee computers these days? I'm guessing Windows. Do they have anything beyond what typical corporate IT does to Windows machines (install virus checkers, auto updates, most users don't have root)?

Clearly that's not sufficient. It sounds like you want some kind of strict compartmentalization like Qubes. There's probably no reason that an e-mail client like Outlook needs to share any state with whatever app they used to manage accounts. Besides perhaps sharing a clipboard for cutting and pasting a tiny amount of info.

The machines probably need secure boot and attestation of the root file system state too. It's pretty bad that in this attack and I think in the Anthem case that attackers were inside their network for such a long period without detection.

I also remember a DEFCON talk where a penetration tester said the hardest site he ever worked on was where they had a strict "star" network topology. None of the computers in the enterprise could talk to each other or even see each other. All communication had to be proxied through a central hub, which would audit all the connections.

Do any banks do that now? Is there any reason they couldn't in practice? I imagine that there isn't really a need for two tellers in the same office to be sharing files directly with each other. Let alone tellers in different offices. I've never worked at a bank do I have no idea what their networks are like. Possibly there would be some uptime concerns with a centralized system like that.

I'm just brainstorming and wondering if anyone has direct work-related experience.

[1] https://qubes-os.org/

[aidos]

It's just staggering.

I know it's silly to think that banks would be better than anyone else, but good lord, malware running on machines capable of transferring millions of dollars that's able to send out video feeds from the network without anyone noticing?! Your various IT/Security teams should be absolutely ashamed. And then the banks don't even have to stand up and admit their incompetence publicly; that's a total disgrace.

That's the state of corporate security I guess. I've dealt with corporate IT departments over the years where they put these "processes" in place to mitigate these security issues but it's all a load of rubbish. Filling in forms to tick boxes so that everyone can go home happy pretending there's security going on, when really their network is a leaky sieve.

At one point I saw a release by a 3rd party supplier to a large corporate system that included privilege escalation, blatantly, at the start of a T-SQL script. It was done because the IT department refused to carry out the action on request via the official channel but it was work that needed to be done to complete a project. The 3rd party knew the admins would just be running scripts as SA so they escalated their own account to do what they needed to do later.

I know it's silly to be so frustrated about it, but we've all dealt with crappy banking systems for years, with totally insane security measures; meanwhile hackers can just walk away with millions using a bit of malware.

[dredmorbius]

A key differentiator for banks vs. many other service providers is that financial transfers can be reversed. Releases of information however cannot be.

So where a bank has a risk of an unauthorized financial transaction, there are multiple options to claw that back (or to shift the risk to other parties, notably merchants).

A disclosure, though, of account information is a different case, and here the results can be damaging to the banks and their customers. One instance I'm generally aware of is an increasing number of disclosures pertaining to offshore banking, many uncovered by the the ICIJ (International Consortium of Investigative Journalists: http://www.icij.org/) and the Guardian. Again, the case involves banks, but it's rather more difficult to reverse transactions when it's your client list and balances, or communications, which have spilled.

Many revealed by insiders, as it turns out.

[aidos]

That's an interesting point, though it's a pretty thin silver lining on a very dark cloud. Being able to undo the operation doesn't really soften the blow of having hackers inside your bank sending outgoing video feeds of employee's screens.

Do you think they'll be getting back the money in this case? Presumably the people involved know enough about the operations to have moved the cash to somewhere out of reach before being exposed.

[vanzard]

> financial transfers can be reversed.

Not this time: hackers withdrew some of the money from ATMs.

[dredmorbius]

Fair point. As the responses note, the damage here is usually limited -- ATMs carry only so much cash each, and (usually) only dispense up to a few hundred dollars (or equivalents) at a time. There've been some exceptions where an exploit is found and utilized in mass effect at many locations in a short period. That takes a high level of organization though.

[ams6110]

The total amount of cash and the per-withdrawal limits in an ATM limits the loss there. You can't steal millions of dollars from an ATM.

[tedivm]

These guys stole $45 million from ATMs- http://www.nytimes.com/2013/05/10/nyregion/eight-charged-in-...

[ams6110]

On Feb. 19, cashing crews were in place at A.T.M.'s across Manhattan and in two dozen other countries ... Starting at 3 p.m., the crews made 36,000 transactions and withdrew about $40 million from machines in the various countries in about 10 hours

I stand corrected. That's quite impressive and amazing that they had that many people involved and nobody tipped it off.

[chubot]

So are you saying that bank IT is no better than corporate IT? They don't have any special software or policies? (like the star network thing I mentioned)

I would honestly expect it to be a bit better than average. I suppose there are many different types of banks and they all vary. Let's just consider your chain banks like Wells Fargo or BoA, since I'm sure somebody around here has worked at one of those places.

[busterarm]

I used to work in financial IT.

Across the industry it's generally better than corporate IT. A lot better. However, it varies widely by sector.

Companies with trading floors or that interact regularly with traders have the best IT practices in the industry. Banking conglomerates are kind of messy - they combine IT operations for each business and never change anything and the systems don't cooperate.

I remember one such company's backup procedures. At that point they were made up of 13 separate large (regional/national) banks. They were trying to standardize the backup procedures between all the banks and run them from a centralized system. At the time that I got there, the nightly backup process failed every single day for over a year and a half. I didn't even get a computer or working logins to be able to do any work for nearly a month. Anyway, getting it to work involved getting the people responsible for the backups at each individual bank's IT group to get their system to cooperate. All of them knew that this would be putting them out of a job at the completion of the project, so there was tons of resistance and it usually took a week of calling peoples' bosses to get the work done. This was also in the middle of forced relocations for most of them. Most of the folks responsible for the work quit. It was really ugly.

[aidos]

To be honest, I have no experience - so I can't say, if the description in the article is at all accurate though, they're in a pretty bad way:

"The cybercriminals sent their victims infected emails — a news clip or message that appeared to come from a colleague — as bait. When the bank employees clicked on the email, they inadvertently downloaded malicious code. That allowed the hackers to crawl across a bank’s network"

There must be plenty of people on HN with experience in this field, so it'll be interesting to hear their take on it.

My (very ranting/rambling) point was that I've seen other large organisations pretending to do security (and probably believing it themselves), where it's really just security theatre.

[linuxydave]

>Filling in forms to tick boxes so that everyone can go home happy pretending there's security going on, when really their network is a leaky sieve.

I saw a DefCon video where the guys were talking about something similar. Lots of small banks in the US use 3rd party services for their banking software. One of them had horrendous security and so some hackers made off with several million dollars before anyone found out.

[burneraccount]

Six years ago I was an intern at a Wall Street firm for a year. The firm that I worked for used an account system that was built in the early 90s and relied on all employees learning special terminal commands to access anything. I can't really go into detail for fear of being sued, but suffice to say the system was archaic. I was amazed that a multi billion dollar company relied so heavily on and invested so little in something essential to the business.

The IT department seemed to use the following logic to justify it: the system served its purpose, the legacy employees already knew how to use it and the developers who made it were long gone thus it was cheaper and easier to just leave it be. While my firm had plenty of developers that could rewrite the system from scratch, their attention was devoted solely to money making endeavors like trading platforms and client facing projects.

As for the "Enterprise Architecture Group" (ie the developer department) that I interned in, the big problem was the heavy reliance on third party development companies. While the firm wanted to hire more developers, simply put very few developers want to work for banks (it's funny though that people in finance would have killed to work at the firm). It would take 6 months to a year on average to fill a developer position and they would have to pay a big premium over the average dev salary with a large yearly bonus.

In order to keep up with all the various projects, they would pay third party development/consulting companies millions to come in and create apps. While this allowed the firm to get the necessary apps "done", it created the most crazy spaghetti architecture you could ever imagine. All these different apps were built using different companies/languages/platforms/technologies then thrown together in a big mish mash of iframes and duct tape. The fact that any of them were able to communicate with each other at all was a miracle. I don't actually blame the developers themselves for this, they would constantly voice their concerns while the completely clueless department head/"architects"/project managers/business analysts would shoot them down. They would say things like "I understand your concerns but Super Consultancy X says that they would do it and it will only take 12 to 18 months!! They are even available to help support the app once its finished!!". Security and use experience were not even on the company's radar, only making money.

[contingencies]

So what defenses should an organization employ to prevent these types of attacks?

Infrastructure architect at a major Bitcoin exchange here.

It's about defense in depth. Processes. An architecture level stance like "do not trust the client, the server, the network, the data center, the hardware provider, or any particular stage within those three elements". Each element validates the other. An alarm raised by inappropriate behavior at any point will shut down an entire instance, cell, or data center before allowing an attacker a foothold.

The only way to realistically take such a stance without going broke or becoming functionally paralyzed is infrastructure level automation beyond what is common in the industry. Hence, cue for meaningful cloud infrastructure management systems spanning private and arbitrary third party infrastructure. Docker-level stuff is about 1/2 way, what we really need is a few degrees of abstraction beyond that.

[chubot]

So, I'm specifically asking about protecting employees' machines. My reading of the article is that the attackers got a foothold on employees' machines and credentials, and just piggybacked their malicious transactions along with normal transactions.

In that case, it doesn't matter how much security you have in your data center. Employees need access to central systems to do their job, so client security is paramount.

For instance, for a bank or Bitcoin exchange, I think it would relevant how many client operating systems can access your crown jewels. I think if you're just using Mac or Windows with antivirus or whatever, there's already a pretty low upper bound on your client security and thus your overall system security.

What I'm wondering if anybody is deploying some kind of custom client OS similar in spirit to Qubes OS, or a build of Chromium OS or Android, which have application sandboxing beyond what stock Linux, Mac or Windows have.

Also, I would imagine that each teller has their own credentials, and the bank should have policies about the transaction rate / total for a single teller. It sounds like the attackers would have to compromise multiple employee accounts to steal that much money. So you also want to protect employees machines from each other as much as possible (not just "outside" attackers).

I'm guessing that a Bitcoin Exchange doesn't have that many employees, since the whole industry is new. You probably have people just accessing stuff with their personal MacBooks or whatever, and that's fine for now (there are bigger risks). But when you start to have 100, 1000, 10,000 employees capable of doing financial damage, then I think this type of thing will start to matter more.

EDIT: Actually I remember one large deposit I made required three people at a bank to approve it. The teller said, "Wait my boss has to approve this." Then the boss said, "Wait my boss has to approve this". So they are probably using the presence of three credentials and credentials at a sufficient employee level to authorize large transactions. So I take it the attackers would have to target employees with those credentials.

But that can cause problems for customers -- e.g. if the branch manager isn't around, you might not be able to do what you wanted. To some degree, they are using meat space protocols to mitigate risk that their software systems can't handle.

[justincormack]

Even widespread two factor auth would mitigate a lot of this. Banks are often quite backward because there are few software suppliers, and it is an industry that took to computing early so there is a lot of legacy. But they vary a lot - the implication of the story is that these were perhaps banks in smaller countries - the banks that got defrauded recently in another large case with cashpoint withdrawals from fake cards were middle eastern. You have a lot of choice of banks, choose the weakest...

[chubot]

I don't believe that's true in this case or in the case of many client attacks.

If you have two factor auth, the employee will go through the process since they need it to do their job for 8 hours a day. Then they will have credentials on their machine (in memory or wherever).

Any attacker sitting on the machine can use those same credentials. Whether you have two factor auth or not doesn't matter.

The point is that you need to prevent the client from getting infected in the first place (which isn't easy if you have 10,000+ employees). As mentioned, if the state of the art is Windows or Mac + antivirus, then your upper bound on security is pretty low.

I recommend reading "Kingpin", a recent book about Max Butler. There's a nice story where he is hired for a penetration test. He guarantees 100% success rate, since he's always been able to get in.

He was coming out of jail and his skills were perhaps rusty, and he couldn't get into this particular server.

So what he did is hack an employee's home computer, steal their VPN credentials, and hack the company server with internal access. Apparently the company was agnry that he did this, but it pretty vividly illustrates the point.

I recall that Kevin Mitnick also used employee VPN attacks. Just because you have hardened Linux, regular updates, jailed processes, etc. on your server doesn't mean it's secure. Employees have to access systems to work, so that is often the weakest link. It's not surprising that this is how major banks got hacked and relieved of millions of dollars.

[ukigumo]

Hi, IT Architect with a history of several major financial institutions here.

Defence in depth is a placebo. Separation of concerns, principle of least privilege, honeypots, SIEM, file integrity monitoring, host intrusion detection, IDS/IPS on all your ingress and egress points, WAF, content filtering and a responsive and empowered SOC capable of acting on auditing events will get you half way to not showing up on the front cover of NY Times.

The problem is that it takes money to keep money safe and too much security is often not secure at all, so putting everything together in a way that you doesn't motivate your users to find new and exciting ways to bypass your controls is an art in itself.

Would love to discuss some of these things with you, any chance I can interview you for my blog?

[contingencies]

Agreed, there's no silver bullet. However, I don't think spending money to feel better is much of an alternative. It feels to me as if internal process design in security-conscious organizations is probably more important than actual systems design... which could be summarized as knowing when not to take shortcuts. Please do get in touch, I'd be happy to chat. Email in profile.

[belorn]

The best defense an organization can employ, is to make departments/managers/people economical liable. This result in insurance being bought, budgets assign to risk management, and practical prevention mechanism being implemented.

No organization like being attacked, but any defensive measure that cost money will always be balanced to the potential loss, risk, and convenience of employees. If the risk feels low, the potential loss minimum (worst case, government will intervene), and employees inconvenience high from employing effective security schemes, then no such efforts tend to be used.

[DominikR]

I'd like to see the sysadmin or programmer that is willing to take the loss if someone hacks the network (or an app) of his employer and steals a few hundred million dollars.

[ams6110]

Professional Engineers (mechanical, civil, etc.) are exposed to liability for the buildings, bridges, etc. they approve.

[smnrchrds]

But we are not liable as long as we follow standards, e.g. building codes. And it's easily verifiable by the government, the employer and the engineer himself whether the standards are being complied with.

Until you have similar standards for software development, I cannot see how such liability shift could work. This is one of the reasons I tend to avoid using the phrase software engineering. It's so different from traditional engineering that it feels incorrect to put it in the same category.

[TwoBit]

It's not enough to put standards in the software development. Users can misuse software regardless of how well it's written. Same as if you build a bridge and users overload it.

[foldor]

Engineers are not liable for not preventing sabotage (like a a bomb for example) however.

[dguido]

The weakest link was that the computer with access to $10 million+ had access to the general web and was running a general purpose operating system at all.

You don't need Qubes to secure this situation. You could use an iPad/Chromebook or a filtering proxy (whitelisted websites) and either would be sufficient.

[TwoBit]

That seems to be the fundamental engineering flaw here. Also, their email system shouldn't allow executable attachments. The last company I worked at completely stopped all such virus infections by killing all executable attachments.

[dguido]

That works too! Why any of this software was running on a machine that can transfer $10mil+ is beyond me.

[xorcist]

You are never safe from targeted attacks but that doesn't mean you should run exotic architectures or give up. There's no reason not to stop carpet bomb attacks. Most of the big known breaches have resulted from those.

Don't run Outlook and don't autorun USB. That should stop most automated attacks, including all the big known ones that breached large companies such as RSA and Google.

To stop the rest, don't surf from sensitive machines, and require two factor auth such as Yubikeys or RSA dongles to log in to them.

Compartmentalize sensitive information on separate machines and networks, and externalise sanity checks of data transactions where possible.

[ryanlol]

PLEASE do not link to qubes in any security related discussion, the devs are known for their incompetence and making some rather hilarious public claims[1].

Not only that, Qubes uses a vulnerable git version from several years ago so practically anyone could go backdoor it if they cared.

[1] http://en.wikipedia.org/wiki/Blue_Pill_%28software%29