Hacker News new | ask | show | jobs
by billyhoffman 4337 days ago
There is much that could be done to improve this announcement:

1- What does "encrypted, salted passwords" mean? MD5 with a static salt? Holy shit, that's a problem. bcrypt? Less so. I have no context to know how concerned I should be, or any indication of how incompetent, or awesome, Mozilla's existing processes and defenses are. Fail.

2- They talk about a "data sanitization process" failing, but then talk about a "database dump file" being publicly accessible. Say what? This could mean anything from "an input validation error allow wrong passwords to work" to "we do a regular database dump, and store that on a public HTTP directory for some cron job to grab." Without explanation, I assume the worst. Fail.

3- "While we have not been able to detect malicious activity on that server..." Again, without the context of what happens, this statement is worthless. If you leaked the database of your users, I won't expect any malicious activity. An adversary wouldn't attack Mozilla. They would crack the passwords of the users and attempt to hijack their accounts on other sites that matter, like, banking or ecommerce sites. At best Mozilla knows this and just wanted to include some proof-point that at least they have logs/basic monitoring of stuff in place, and wanted to save face. At worse, Mozilla truly believes that someone not actively attacking them somehow means that nothing bad will happen from this loss, which is stupid. And Mozilla's Security usually isn't stupid. Fail.

4-" In addition to notifying users and recommending short term fixes, we’re also taking a look at the processes and principles that are in place that may be made better to reduce the likelihood of something like this happening again." This is a completely unsatisfactory statement. If you just discovered the problem this afternoon, like, "oh shit, why is the a .sql dump in our HTTP readable /backups/ folder?" then saying "hey, we discovered a problem, we think we have stopped it, and we are looking into our processes" is a reasonable response. However when you have "just concluded an investigation" you should, I don't know, tell us your conclusions maybe? What happened? Why did it happen? What changed in your existing system that allowed it to happen? Or has this short coming always existed? If so, who is defining/vetting your processes? What are you doing so this issue doesn't happen again? What other thing are you doing to watch the thing that's going to make sure it doesn't happen again? Instead, we get a generic statement. Fail.

While not as completely opaque as some "oh no, we got pwn3d" posts, this blog post has completely failed to do the 3 things any post of this kind should do: 1) educate me about what happened 2) help me understand the risk Mozilla's actions have exposed me to, and 3) give me confidence by demonstrating clear actions you are taking so this won't happen again.

Yes attacks happen, but when a company or organization is up front, honest, and over communicates, it does wonders to calm the situation.

Mozilla, I expect more from you.

4 comments

A process failed, and the DB dump that is published to help contributors improve the MDN site got out unsanitized. The sanitization/publication process will be redesigned to include stricter controls. For now, it is shut down.

MDN has been using persona for a while now, meaning that most accounts don't have passwords in the database. But older accounts still had the SHA256 salted hash that Django creates.

We traced back as much as we could. Access logs, netflow data, etc... We found that the tar.gz containing the DB dump had been downloaded only a small number of times. Mostly by known contributors. But we can't rule out that someone with malicious intentions got access to it.

Who exactly are these "known contributors", and why did they have access to this data? Why did they not report the problem earlier?

And if it was downloaded "mostly" by "known contributors", who was involved with the rest of the detected downloads?

https://bugzilla.mozilla.org/show_bug.cgi?id=932869 was the request for a sanitized DB for folks wanting to develop MDN itself. We could identify most of the handful of IPs that downloaded the file during the time period where it was unsanitized to individuals (i.e. IPs inside Mozilla offices, etc.). However because some IPs were unknown, or public, or potential NAT addresses Mozilla decided it was best to disclose the issue.
If some of the accesses were by people or systems within Mozilla, can you please address why a month went by before the problem was noticed?

If there was enough need to justify putting forth the effort required to export a sanitized version of these data for developers to use, then why didn't these users notice that something was wrong much sooner? And if they did notice, why weren't the appropriate parties within Mozilla notified sooner?

Could you please provide more specific details about these IP addresses that couldn't be accounted for, too? Perhaps a list of them, for instance? At least then affected users will be able to make their own call regarding their level of risk due to this incident.

Sorry, I can't provide a list.
Why not?
Sha256+salt. See https://github.com/mozilla/kuma

We are still working on the rest.

Why did you decide to use sha256 instead of a kdf like bcrypt or pbkdf2? I'm not attacking you, genuinely curious.
To provide a bit more context, in early 2011 we made a conscious decision to move towards the password storage methods described here : https://wiki.mozilla.org/WebAppSec/Secure_Coding_Guidelines

While we were moving in that direction (upgrading apps, etc) we also launched Persona (BrowserID at the time). Some apps opted to switch to Persona, others opted to upgrade password storage mechanisms.

That's an interesting document.

It mainly contains assertions for what one should do. Do you know if there's an explanation for the rationale anywhere? For example _"Passwords must be 8 characters or greater"_ or _"Privileged accounts - Password for privileged accounts should be rotated every: 90 to 120 days"_.

Not on hand, those decisions were made years ago, and done in email discussions and in person meetings.

That said, the password length requirements were driven by the cost of performing effective brute force attacks against properly hashed and salted values at the time we set that length.

Privileged passwords was basically a stop gap measure to ensure that users were refreshing passwords regularly. The correct solution is to deploy multi-factor authentication.

It's what django uses, and this site uses django.
Django 1.3 or lower. Django uses PBKDF2 [1] since 1.4 (March 23, 2012) [2].

[1] https://docs.djangoproject.com/en/1.5/topics/auth/passwords/...

[2] https://docs.djangoproject.com/en/dev/releases/1.4/

MDN was on Django 1.2 for a while, albeit with monkeypatched password hashing (since Django at the time was still defaulting to SHA1, I believe). With the switch to Persona, it no longer matters -- for a new account -- what hasher is used, since Persona doesn't involve storing a password.
Not to mention it's always been easy to implement even when it wasn't bundled by default.

There is utterly no excuse for storing passwords with anything that's not PBKDF2, bcrypt, or scrypt starting in 2009.

Right, they switched over to Persona, which is far better. Unfortunately the old hashes were still left in the DB.
also the feature was built a long time ago.

current standards for sites not using persona are here: https://wiki.mozilla.org/WebAppSec/Secure_Coding_Guidelines

As the number of logins per core increases the login latency with these functions increases fairly quickly.
Please try also to improve the page load time by using a cache, faster hardware, etc.
This was one of the least detailed and helpful disclosures I've ever seen.

Reading "encrypted passwords", and then "salted hashes" without any specification of the hash algorithm from a tech company like Mozilla is utterly astonishing to me.

I feel like whoever discovered this breach sent it over to the PR department without any sort of final review of the release by an engineer.

Not at all, the decision was made to release the information necessary to help people decide if action was needed. As you can see from this thread, we are happy to provide relevant details, and like most of our other tools, Kuma, the platform that MDN uses is open source, and on github.
>What does "encrypted, salted passwords" mean? MD5 with a static salt? Holy shit, that's a problem. bcrypt? Less so. I have no context to know how concerned I should be, or any indication of how incompetent, or awesome, Mozilla's existing processes and defenses are.

Given that this leak happened at all, incompetent to criminally negligent would be my estimation.