[VWWHFSfQ]

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts.

What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?

I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.

[majormajor]

The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.

[oefrha]

For any individual/organization remotely competent at security (i.e. not using Passw0rd! as password and reusing elsewhere), a leak of hashed password is probably the least of concerns. Compared to this anyway:

> According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.

[glenngillen]

former Heroku employee, though long long long ago, with no specific knowledge about this incident, but...

We did so much work in open source it was just easier to assume everything was always publicly viewable, or that what you were doing now might be open sourced in the future along with the full commit history. Whether something was private or public was more a business decision around competitive risks and not a security-led one. To that end I'm far more concerned about a database and passwords getting popped.

But who knows, a lot can change in 10 years. Maybe private repos being exposed is also very bad.

[gabereiser]

This is also why folks advocate for a separate auth system (keycloak was just mentioned here on HN) since they are different types of information.

>I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.

I feel like they are heading this route as well. Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.

[huijzer]

> Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.

Trust takes long to build and is easy to break. For anyone able to convert the Heroku buildpack to a Docker and able to move the database, moving away from Heroku shouldn’t be too hard. There are multiple similar services nowadays.

[throwawayboise]

Yep. Same thinking lead to the split of /etc/shadow from /etc/passwd.

[donatj]

We moved basically everything but username into an entirely different db, went so far as to hash the username column so we don’t even know what’s it is until you log in.

[3np]

> hash the username column

Is this giving you any real security benefit? (I'd assume the usernames are indexed elsewhere and that it's a reasonable assumption that whoever gains access to this hashed data has access to the username list as well, making a lookup trivial - or are these not safe assumptions?)

[couchand]

Not GP, but one architecture where there would be a difference is there's a distinct identity provider, responsible solely for exchanging the user's long-lived username/password credentials for a short-lived ticket.

[bigDinosaur]

I imagine the process going a bit like https://youtu.be/y8OnoxKotPQ

[christophilus]

I knew what that was before I clicked it. Never disappoints.

[donatj]

It's a PII covering our butts thing more than anything. Wanting to absolutely minimize what we know about our users.

[glenngillen]

Yeah, the comms around this has been very concerning. Do I need to rotate every config var on all of my apps? Re-install every add-on?

While the nature of what limited things they had disclosed to date pointed to this situation part of me wanted to believe it wasn't as bad as I was assuming. And now the trendline on this suggests I should have already done everything I've outlined above. And I'm low confidence anybody is going to be proactive in telling me until it's absolutely obvious that these things have been compromised and exploited.

[xena]

IIRC the environment variable settings are encrypted in a physically separate database. However it may be a good idea to rotate your secrets anyways. My hunch would be that there are so many "juicy" targets on Heroku that you probably don't need to worry too much right now unless you are or work for a "juicy" target.

This is gonna suck.

[t_sawyer]

I posted a similar worry about the ENV's.

Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.

[benevol]

> What else was in this database? Typically the password field is stored alongside the rest of the user record.

That is where you are supposed to have encrypted as many data fields of the record as possible - in addition to the conventional database encryption which encrypts the database as a whole.

[jeremymcanally]

The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase.

If that's the case...yikes.

[jeremyjh]

They didn’t say that happened. I’m reading it as their DB was compromised and it’s contents included GH auth tokens.

[jeremymcanally]

How do you read this as the database had tokens in it?

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database...

EDIT: Ah yep you're right. Two tokens in play there: one Heroku API token, one GitHub token. Phew.

[sofixa]

> I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up

They can't, they surely have EU customers so have to follow GDPR disclosure rules, which they might already be afoul of.