[lovelearning]

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation.

Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku?

I'm not asking this rhetorically or in bad faith. It's a genuine question I have based on a project I did. I researched cybersec tech like SOAR, XDR, security logging, and SIEM in depth. On paper, the marketing for such tech gives the impression that by using them, such breaches can be detected and prevented in real-time. But there seems to be a mismatch between the claims and ground realities. If so, why?

[Mandatum]

Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding).

In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "security" ends looking the same.

They don't pay well, executives have pulled talks and fired speakers who do things they disagree with (the same executives are promoted and remain there - no accountability), they've got a pretty bad wrap within the industry.

[lovelearning]

Let's assume that prior to acquisition, Heroku sec had set up a very secure posture using such tech. Then they lost most of their experienced people after acquisition.

Some questions:

1) Are these tech not enough to enable others - perhaps less experienced, or experienced but not on a particular product - to take over while maintaining the same posture?

2) What kind of additional (perhaps intangible) security does an experienced team add to the posture that gets lost when they leave?

3) As I understand them, things like risk frameworks, NIST CSF, security assessments are all supposed to anticipate people problems (resignations, malicious insiders, etc) and make the posture as independent of them as possible, probably relying on automated tools like XDR and SOAR to do their thing regardless of who's sitting at the console. Does it not work like that in reality?

Btw, thank you for your reply and insights (and to everyone else who replies)! Pardon my probably naive questions. I'm an outsider looking in and having trouble understanding this phenomenon of data breaches in the face of all the tech marketing.

[milkshakes]

Fundamentally, a security analyst authors detections, reviews surfaced alerts, or identifies hypotheses to investigate. In the case of reviewing surfaced alerts (the firing of a detection which may or may not be authored by the security team), differentiating true positives from false positives is subtle and often requires context or further digging. Of course, this requires time, which costs money, so you can imagine the tension there.

This process can often be subtle, and difficult to automate. In many cases, the issue is automating the economical delivery of enough context to the deciding function that a clean choice can be made. However, even with enough context, and enough documentation, escalating vs suppressing an alert can often be a judgment call. Humans are meat based pattern matchers, and a decade's worth of "ML" and "AI" advancements still not sufficiently precise (as in vs recall) enough to filter out "things that look bad" from "things that are bad, for our specific environment", that knowledge still lies with the security team.

[muraiki]

Only the slides are available, but the presentation "AI is Not Magic: Machine Learning for Network Security" at CMU's FloCon in 2020 was about this: https://resources.sei.cmu.edu/library/asset-view.cfm?assetid...

[gruez]

>bad wrap

https://www.merriam-webster.com/words-at-play/usage-bad-rap-...

[dataspun]

indeed, unpalatable

[richardfey]

> fired speakers who do things they disagree with

Can you please make an example? I am genuinely curious about what things

[intern4tional]

https://tech.slashdot.org/story/17/08/10/1919204/salesforce-.... is probably the most well known incident. There have been others though.

[Mandatum]

Hi Jim Alkove!

[sfdcsecthrow]

He has retired.

[everfrustrated]

Most security products are snake oil. The vendor selling them and the people buying them (always business execs never engineers) have no interest in how well it actually works as a company is never buying these tools to actually be secure only ever to give the impression to some higher exec within the same company or tick a box on some audit process (which requires something to be in place but doesn't require verifying it works).

It's beyond a joke how the "cyber" industry operates.

[sfdcsecthrow]

Throwaway for obvious reasons. But in my experience Salesforce security org is plagued with incompetent leaders who chase arbitrary metrics that does not improve security at all. At one point in time, security team at Salesforce was stellar and did some awesome work. Dont get me wrong, there are many many smart security engineers still around but their population has been dwindling. This all started when a bunch of new leaders were hired. Instead of promoting tenured smart people, the security leaders decided to bring in their own gang of coworkers from previous employers. At the same time they slowly started pushing out tenured leaders.

These new leaders are typical VPs who have long lost any technical chops and its a huge task to explain any complex technical topic to them. On top of that they don't bother understanding the fundamental business model and just want to push their agenda on to everyone. So they have added "security processes" which requires checking boxes. The more boxes you check the more metric it generates the better the leader looks. These security leaders are so disconnected from the ground reality that they don't even realize that all they are doing is adding hurdles in the path of engineers without improving any security.

[srinathkrishna]

I do remember the whole debacle that happened at DEFCON and people got fired for presenting something on stage.

Agreed, there's a lot of interesting stuff that came out of the Security (or related orgs) at Salesforce. Red team tools, chaos tools and JA3 which we use at my current work as well for SSL/TLS fingerprinting.

[lovelearning]

Thanks for the insight!

[cube00]

> GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation.

The three days after being notified to actually revoke the tokens isn't ideal either. Surely if GitHub comes to you and warns you of suspected unauthorised access you'd spend a very limited amount of time and then revoke the credentials to be on the safe side.

[yuliyp]

I think (based on what I've read on the linked page) the notice was telling Heroku that "hey, someone used a compromised OAuth token to download your source code" not that "the tokens that you are using to read Github repositories of your users are compromised". Both are Github OAuth tokens, but playing different roles. Presumably the compromise of source code might have been used to help get access to the database that had the Github integration OAuth tokens, and realizing that might indeed have taken a couple days.

[adrr]

Because Heroku/Salesforce doesn't have real security. Requiring special characters in passwords and sending out emails that have http and not https links to a password reset page. Their security is a joke.

[nameless912]

As a former Herokai, let me color this a bit:

Heroku _used_ to have their own security team which was quite good and had some scary talented people on it. However, over the last 3 years or so Salesforce has been forcing Heroku to adopt Salesforce's operations practices, and this has not only wrecked productivity but completely destroyed morale and caused many, many of those talented people to quit. I for one decided to quit after only working there for around 8 months due to a horrific overreach by Salesforce into Heroku's operations.

Among other things, Salesforce forced us to adopt:

- their internal ticket tracking system, which _runs in an instance of salesforce_ (barf)

- their slack instance, which lost us many of our customizations and broke a bunch of integrations for weeks (I wouldn't be altogether surprised if this was one of the causes of the delay in notifying Herokai as to what was going on)

- their incident management process, which requires us to notify "Salesforce ops HQ" anytime there's an outage that meets certain criteria.

This last one was especially bad, and meant that we no longer had full agency to act during incident response situations. I had one incident I responded to where the problem got worse while we waited for Salesforce IM to spin up, so that we ended up having what would have been a 10 minute outage turn into a 2 hour outage because the issue got out of control.

In short, the problem isn't the people trying to administer Heroku; they're great folks under a lot of pressure with very few resources. The problem is, and has always been, Salesforce's "leadership" deciding what's best for a cloud platform they couldn't give less of a damn about.

[Rastonbury]

At this point, I'm not sure why SFDC even bought Heroku.. Is there major overlap between CRM users/buyer (salespeople) and Heroku's who are mainly devs, hobbyists and startups (i'm guessing)? Surely they didn't try to buy hero to compete with the big 3 cloud providers?

[nameless912]

Welcome to the existential question of Heroku from the inside :) No one knew what the point of the product was anymore. My vibe is that SFDC's goal was to use Heroku as their cloud provider for everything, and when that didn't work (more due to lack of focus than technical issues), they tried to shoehorn Heroku into the Salesforce platform as a tack on feature. It was all very weird, no one knew what we were doing, and everyone was upset about it. I originally joined the org because I was really excited to help the early startup/small business/hobbyist/student sector, and SFDC writ large just did not care about those customers IMO.

[sfdcsecthrow]

The integration of Heroku stuff into SFDC would have probably been easier if the Heroku folks got off their high horse.

[nameless912]

I don't want to start a flamewar, but it wasn't an integration; it was a hostile takeover. Heroku was doing just fine without SFDC's interference, and when Salesforce not only refused to believe that employees might have negative opinions about integrating but actively prevented us from speaking up about it, people got rationally angry and left. I remember talking with an architect on the Heroku side about how he felt about all of it, and he told me that it wasn't presented to him as a choice, and senior leadership was convinced that Herokai would see this as a good thing despite his (and others') warnings.

[sfdcsecthrow]

This is not true. Heroku actually had a pretty decent security team. They have lost most of that talent though.