[0xUser]

Source (with more details): https://blog.twitter.com/en_us/topics/company/2020/an-update...

> The social engineering that occurred on July 15, 2020, targeted a small number of employees through a phone spear phishing attack. A successful attack required the attackers to obtain access to both our internal network as well as specific employee credentials that granted them access to our internal support tools. Not all of the employees that were initially targeted had permissions to use account management tools, but the attackers used their credentials to access our internal systems and gain information about our processes. This knowledge then enabled them to target additional employees who did have access to our account support tools. Using the credentials of employees with access to these tools, the attackers targeted 130 Twitter accounts, ultimately Tweeting from 45, accessing the DM inbox of 36, and downloading the Twitter Data of 7.

[dang]

Ok, we'll change to that from https://www.reuters.com/article/us-twitter-cyber/twitter-say.... Thanks!

[dpweb]

No disrespect to those challenged with protecting such a huge target, but why do admin tools even have these capabilities? I could see needing to disable a user account or change some attributes, but why would an admin ever need to tweet from it? There shouldn't be tools with God privileges even for admins. Not surprising human error was involved in a breach this huge. So, how many people had access to this tool? Is there a killswitch for the tool itself available to very few, really very few, persons? edit: I dont know if the tool can tweet but surprised 2FA can be stripped without a human being confirming (ie.. the acct owner's social media person), especially for famous people.

[extrapickles]

The admin tool was used to change the email on an account, then the attacker reset the password and got full access to the account. Apparently having 2FA enabled did not stop this attack (admin tool probably had the power to strip 2FA from accounts).

So while the tool did not directly have the ability to tweet, it effectively did.

[csunbird]

I feel like the power to reset emails and remove 2fa should be only held by a very small subset of customer support, with proper training.

[c00ls0sa]

I work in customer service supervising entry level employees. The amount of power they have at any given time is astounding and it's by sheer ignorance or benevolence that more isn't embezzled en masse or this information isn't used for personal gain. My entire team of newly trained staff have access to your bank account information, where and when your payment was posted by IP, and we can strip 2fa or mobile numbers at whim. This coupled with inexperienced agents often leaves multiple accounts compromised. Having a select few engineers who don't work weekends always helps. We don't train the agents to tell them that they could potentially ruin customers' weeks by pushing the wrong button and it happens way too often to be standard, but as long as investors are happy and banks are good to reverse charges with no penalties here we are. Tech companies are good to throw caution to the wind.

[chedabob]

That seems like it was the case, but the attackers got access to lower privileged accounts and used them to find who had that access so they could target them.

[toyg]

The key being "proper training". Those few god-level admins should be drilled enough to defeat a phone-phishing campaign. In fact, they should probably have custom procedures to look after their own credentials.

[f2000]

I've coded/supported a number of admin tools for a number of large companies (banks, telcos, etc) so I'll take a stab at "why" First, god mode can be implemented cheap and fast. God mode also makes day to day support easy as admins/support staff can do just about everything fast and easy - so they typically love it and will often fight to keep it. More than once I have had managers turn down proposals to tighten up security because of the cost and false beliefs that because the tool is behind a firewall, better security is not needed. Taking a schedule and/or budget hit to implement tighter security is not going to get them a promotion or bonus. Sad but true. In the vast majority of cases I've seen, paying for security improvements becomes incentivized for managers after their company gets burned by a breach.

Also, admin tools are often "afterthoughts", there is usually a motley collection of them, and often considered as an expense/cost to be minimized and not a revenue generating asset that gets more budget and attention.

[ameliaquining]

My uninformed guess is that there isn't a "tweet as this user" button (because obviously there's no legitimate use case for that), but there is a "change this user's email address" button (because you might need to do that in order to help someone who's locked out of their account), and if you can do that you can take over someone's account. Obviously something like this would be detected quickly, which makes it less scary in some ways than a "tweet as this user" button, but of course this particular attack did not seek to evade detection once it was launched.

Of course, some of the targeted users presumably had 2FA enabled. How to do account recovery with 2FA in a consumer context is a complicated problem and I'm not aware of any good answers, but there's certainly an argument that the protections in place there weren't adequate and I wouldn't be surprised to see them changing soon.

I would also hope that rank-and-file support staff can't change users' email addresses, and the attackers had to spear-phish one of a smallish number of people whom more complicated account-recovery cases are escalated to. But who knows if that's how it works.

[michaelt]

> How to do account recovery with 2FA in a consumer context is a complicated problem and I'm not aware of any good answers

I've always wondered why there isn't more use of time delays for this sort of thing.

If there was a notification e-mail and a 7-day wait, that would offer a fair chance for the real account holder to cancel the change. Not 100% - the user might be on holiday - but it would catch a lot, and hence decrease attackers' motivation. And while a 7-day wait is inconvenient, for services like Twitter and Steam losing access for a week isn't the end of the world.

[onefuncman]

"Tweet As User" is the basis for almost all social media management tools and Twitter still doesn't have very fine-grained permissions grants. https://developer.twitter.com/en/docs/basics/apps/guides/app...

We had a running gag in our social media startup of tweeting "poop" from people who left their phones/computers unlocked ... someone did it to an employee that was logged in as a customer (corporate brand) context and that was the end of that 'joke'.

[dlkmp]

Did they tweet directly from the admin tools? My impression was that they used the admin tools to reset the password and then take over the account, ultimately tweeting like any normal user would do.

[wraithy]

Do we know for sure that the admin tools can do all this? My understanding was that the tools enabled password resets, which allowed the attackers to tweet from the accounts themselves.

> For 45 of those accounts, the attackers were able to initiate a password reset, login to the account, and send Tweets.

[erk__]

It is my understanding that they used the tools to update the email of the account, then reset the password to log into and make a new password such they could log in and tweet. Do you have any source that says that they could use the support tools to tweet directly from?

[throwaway345346]

Internal admin tools grow over time. They don't spring fully-formed into the modern company with all the correct access controls and auditing they ought to require at that point. They carry a lot of cruft from things that were needed early on and aren't needed anymore.

Furthermore, they're a classic cost center, not a lot of love or budget goes into reducing their tech debt, or bulwarking them up against a sophisticated adversary. Red teaming yourself full time is expensive and not profitable. What's the worst that happens from a breach like that? Well, Equifax is still going strong!

I recall being party to an amusing conversation at a major network services provider at a team meeting for people with access to such tools, to the effect of:

- Alright, we're modifying <internal tool A> to lock down access to accounts related to <major political figure>. You will no longer be able to use <internal tool A> on <accounts>, only select supervisors will have that access.

%%% ah, okay, that makes sense

# uh, hey, regarding <internal tool B>, which allows us to look up <thing that would provide equivalent access to internal tool A>? does that still work the same?

- Uh, yeah, it does.

# okay?

%%% ... silence ...

- Alright, next item!

To the best of my knowledge, that was never addressed. <internal tool A> has audit logs. <internal tool B> doesn't.

[tallanvor]

Admin tools having the capability to change email and 2FA settings is a necessity, but Twitter clearly needs to greatly increase the security.

I don't know what all Twitter uses, but I know that many companies have various methods of authentication depending on how much damage can be done:

- Logging on using a username/password and 2FA is enough for some activities.

- More sensitive operations have to be done on hardware that has a certificate installed and backed by something like Windows Hello.

- Even more sensitive operations require a JIT account and a certificate stored on a separate hardware key such as a yubikey.

- Very sensitive work gets done on a secure device that is very locked down and can detect changes to the hardware that may suggest tampering.

- Some stuff simply isn't allowed to be done remotely, even with the above restrictions.

Obviously not every company needs such a complex setup, but for someone as high profile as Twitter, you'd expect more thought to be put into this.

[geocar]

You have a need for a tool allows you to see the UI “as the user does” so you can respond to support requests and maybe someone thinks it’s easier to just copy the cookie.

This isn’t the right way to do it, but given they work at Twitter I could imagine this isn’t the first big mistake they’ve made.

[Thorrez]

This sounds like speculation. Is there evidence that Twitter has a tool that allows employees to see the UI as the user does?

[geocar]

The question was interpreted as “why do admin tools have these features” on account of that was in the first sentence, and not, as you may have imagined, a request for a twitter employee to explain all of their tools, or a request for a twitter employee to justify creating these tools.

[onefuncman]

it's not copying the cookie, but there are absolutely third party UIs via a user's API key, it's not a huge leap of faith to assume Twitter has similar internally.

[Thorrez]

A tool that allows an employee to access a user's API key? That sounds like a bad idea, especially if that tool is accessible to support personnel.

[Thorrez]

I doubt the admin tweeted from the tool. The admin changed the email on the account, then did a password reset, then logged in as the account then tweeted.

[yipbub]

They probably had permissions to change/reset the access credentials which can be used to gain access as a the user.

[andrew_]

I'd like to know more about these tools. That there's at least one which can bypass a user's 2FA settings without notification suggests that there are additional tools in the same vein.

[ma2rten]

Google requires its employees to use a security key for access to all internal systems including admin tools, source code and email. Every since google started enforcing this policy the number of successful phishing attacks has gone down to basically zero.

[londons_explore]

I believe the Twitter attack involved tricking a user into installing proxy software on their machine (to be in twitter's internal network).

If that is the case, that same proxy software could proxy the security key requests too.

[tdhoot]

That can be prevented by restricting the software that can be installed on employee machines.

[c00ls0sa]

WFH has caused many companies to ease up on restrictions involving location, ip, and sometimes a broader need for software. Granted, nobody should be this easy to bamboozle, but I get why now more than ever this may have been an issue.

[Thorrez]

If there's malware involved I don't consider that phishing. Although some would debate that.

[Thorrez]

Did you reply to the right comment?

[spullara]

Every network has to have tools to do that. How else will they enforce the laws they are required to enforce?

[cmelbye]

Those legal requests aren’t serviced with a password reset in order to log into the account. It seems more likely that there’s an internal tool to help people who have lost their second factor, but that’s just a guess.

[Thorrez]

>without notification

Are you sure there wasn't a notification?

[ehsankia]

Were the spear fishing attacked also used to get 2FA, or did these accounts not have 2FA? Would hardware based 2FA not have stopped this?

[technion]

Without details it's hard to say. They talk about "phone spear phishing". It's within the realm of that description to say they were hit the same way someone I know recently was - someone called and said "Can you just install Teamviewer on your desktop for me and give access to a logged on session".

[lighthazard]

Maybe? I imagine a spear fishing attack could entail the target is sent to a false panel to login where it sends a true 2FA request. The target then freely gives this 2FA code to the attacker.

[Thorrez]

That compromise OTP 2FA, but not U2F 2FA.

[bananabreakfast]

It says right there in the linked page that the phishing attack was able to manipulate multiple employees into giving access past their 2FA.

[Thorrez]

Yeah, that answers 1 of the questions. But we still don't know what type of 2FA was being used, or what technique the attackers used.

[zamalek]

I haven't seen a form of phishing that hardware 2FA doesn't stop. Yes, it would have.

[monoideism]

Significant forms of phishing are stopped by U2F (as used by Yubikey and others), by crytographically binding your credential to the domain name , and only issuing an approval signature if the site matches. Obviously, this stops credential phishing.

https://krebsonsecurity.com/2018/07/google-security-keys-neu...

https://medium.com/@antonisikora/how-u2f-security-keys-can-e...

[Thorrez]

You and zamalek appear to be in agreement.

[monoideism]

You're right - I misread the comment. I'd delete my comment if I could.

[Thorrez]

It does depend on whether the hardware 2FA is U2F (which stops phishing) or OTP[1] (which doesn't).

[1] http://www.tokenguard.com/RSA-SecurID-SID700.asp

[fareesh]

The way it is worded can also mean that there were XSS vulnerabilities in the internal tool since they are saying "gained information about how our processes work". I feel like that's a strange and vague thing to say. The right kind of xss vulnerability would enable them to bypass 2fa too, maybe steal backup codes even.

[berkes]

I understood that line as 'saw names, contact detail, positions and permissions of employees'

[Thorrez]

If there's an XSS attack I don't consider that phishing.

[fareesh]

Isn't it both? Phish first user, post to internal tool and xss attack second user.

[Thorrez]

Yeah I guess you're right, it could be like an exploit chain where 1 link in the chain is phishing to gain access to something and xss is the next link for lateral movement.

But I don't know what "The right kind of xss vulnerability would enable them to bypass 2fa too" means. If the attacker doesn't have 2FA I would think the attacker can't log in, thus meaning the first link of the chain has no purpose.

But I also think XSS in this case is not very likely. From interviews with the attackers it sounds like they're social engineering experts who hang out on social engineering forums, not XSS experts[1][2][3].

[1] https://krebsonsecurity.com/2020/07/whos-behind-wednesdays-e...

[2] https://www.nytimes.com/2020/07/17/technology/twitter-hacker...

[3] https://krebsonsecurity.com/2020/07/twitter-hacking-for-prof...

[harikb]

> ultimately Tweeting from 45

for a moment I thought it read `tweeting from 45's`

[DoofusOfDeath]

Nah, you're thinking of putting a 33 RPM disc on the phonograph, then setting the playback speed to 45 RPM.

Hands down the easiest way to make Shaun Cassidy sound like one of the Chipmunks.

[nodesocket]

Why are internal employee tools publically accessible? Minimum they should require VPN access, but really go further with Zero Trust.

[whoisjuan]

AFIK, in a Zero Trust Architecture a VPN is considered a perimeter and therefore it becomes a vector of attack to access systems of authoritative decision.

Many security researchers have already established that the benefits of a VPN especially in the modern distributed world are marginal at best.

Basically, yes a VPN makes you a tiny bit safer but it also adds a lot of networking complexity and adds more friction to the job of your employees. It also becomes an attack vector for malicious parties, since once they get VPN access they can theoretically access at least the first layer of protected resources.

So in layman's terms an attacker just needs to phish for VPN credentials, maybe steal an OTP token and they will have access to a non-trivial amount of network protected resources.

On the other hand if every service you use has its own authentication then the attacker needs to target each service and to know what services to attack they need knowledge that is possibly contained in another system that also requires authentication and is definitely not guaranteed for the attacker that all the systems will have the same password and/or have 2FA disabled.

Honestly, in my opinion VPNs are just an excuse to monitor traffic. This is a bit of cynical take, but I'm convinced that companies that use VPNs are more interested in seeing what goes in and out their network than in protecting their resources.

[jcims]

Depends on what you're defending.

If your enterprise is a global network with millions of nodes operating a blend of modern and legacy systems accumulated through hundreds of acquisitions in 100+ countries over the course of the last 50 years, a VPN with hardware tokens isn't a bad additional layer. It isn't even mutually exclusive with zero trust, it's just another layer of auth and access.

Twitter? Largely a different story and commando zero trust might be a viable option. As observed many other places, this sounds like a poor authentication model and probably poor governance for highly privileged access. Presumably they will take a look at their authentication, which sounds like it's making some bad assumptions, and improve.

[AnonC]

> On the other hand if every service you use has its own authentication...

This would be a nightmare for the people managing any nontrivial system. There are good reasons to use something like Active Directory and tie systems and applications to it for easier policy enforcement and management. There are good reasons to avoid this centralization for certain things too. Either extreme would be an exercise in frustration.

[whoisjuan]

Certainly. That’s why things like Okta make sense. It allows people to use it as a Password Manager while keeping certain level of sanity in managing resources but without giving up individual authentication against services.

I’m not so sure that it works that well once it becomes the actual authentication middleware. But as a single sign on directory it definitely reduces the complexity for the employees and for IT departments.

Either way I think more than systems, people need training. I know there are sophisticated phishing attacks but someone who has been trained to understand and acknowledge these situations should be able to detect when someone is trying to steal information.

I think Twitter’s failure was to not properly train their employees especially when they are such a visible and juicy target for bad actors.

[gsich]

>Many security researchers have already established that the benefits of a VPN especially in the modern distributed world are marginal at best.

Yes, with the (wrong) assumption that after you have connected to a VPN, all other services are free for the taking, without any further authentication.

[whoisjuan]

I have worked at companies that used VPNs. After you authenticated and logged with the VPN you had access to several resources with no further authentication. Granted I would always use the company issued computer so I don't know if there was another non-transparent authentication in the background but overall seemed that just being within the network was enough to access things.

[Hikikomori]

There's a lot you can do with vpn to make it more secure.

On our vpn we require a non-exportable certificate in the tpm chip, normal user credentials, then we have a captive portal that forwards to our SSO that requires a yubikey.

[sah2ed]

Do you have a writeup describing your architecture in more detail?

[bawolff]

The blog post is vauge but definitely implies that a vpn was in place.

[Thorrez]

What if the attackers phish the VPN credentials too? Does Zero Trust imply phishing-resistant credentials? What Twitter needed was phishing-resistant credentials (security keys, aka U2F).

[staticassertion]

Zero Trust != VPN. Zero Trust means that the network is not what determines trust.

Consider this: * You go to your office, connect to the network * Now you have access to internal services, by virtue of being on the network

In a Zero Trust network it does not matter what network you are on. Trust is handed out individually, based on the identity/ role of the user and the context of their session (is their os patched? running security tools?).

[Thorrez]

How does the site know the user's OS is patched? The User Agent? How about whether security tools are running?

The attacker can surely use a patched OS. Are the security tools secret? If not, then the attacker can run the security tools too.

[staticassertion]

> How does the site know the user's OS is patched? The User Agent?

User agent is a great place for a version 0, sure. 99% of your assets aren't compromised, so worrying about a bypass isn't important to most of them. For a v0 just knowing that most of your boxes are patched is a huge win.

Of course you'll want client certificates on devices, or some sort of TPM, which is how Chromebooks work. The attacker having a box is not enough - identity is a key principal of zero trust networks.

[ThA0x2]

Another instance where zero-trust networking has utterly failed.

Security comes in layers. That first layer of requiring a VPN can stop many types of attacks from happening.

Next layer is requiring MFA for VPN access. Then for admin access, require MFA only from approved devices on the domain.

Large banks and the DoD have been doing this for years.

The "fail often and fail fast" crew are always reinventing the wheel after bad experiences. I honestly feel sorry for them.

[dcow]

? This definitely wasn’t a zero trust failure.