[freejack]

tl;dr: guy from hover, mea culpa, new code on the way.

I thought it might help to provide some further deets on that blog post.

I don't think we're making a case there, or providing an excuse - it certainly wasn't my intent to try and convince anyone of anything when I wrote that, but rather, it was an exercise to explain where we were (with that and other development projects) and where we were going.

We've gone back and forth on how we handle passwords over the years - and it has always come down to what type of interaction do we think will be best for our customers. I haven't re-read that post from April today, but I think I mentioned our last go-round on this made it much easier for our customers and customer service people to help in bound callers and sacrificed too much in terms of security.

We'll probably continue to go back and forth iterating the implementation, each time narrowing the swing of the pendulum until we find something that more appropriately balances what we think our customers are looking for in terms of security and usability.

I'd also like to point out that the scope of the risk isn't trivial. For example, URL-based password resets are only as secure as the mailbox they are sent to. i.e. a significant number of domains are stolen and threatened to be stolen through email account exploits (re-registering previously used addresses, forwarding attacks, etc.) This is made even more complex when a domain expires and email on that domain stops functioning. Where should the password reset go? We get dozens and dozens of calls a day from people in this position that need our assistance, making it tough to simply send out a reset request.

Anyways, I didn't come here to make excuses, I just thought I should acknowledge that we're aware of the gap (we caused it!) and working on it and considering the whole set of variables. Security is our primary consideration but that doesn't give us the luxury of ignoring the usability implications. Were that the case, we'd simply issue two-factor fobs to our clients and be done with it.

And finally, to those of you that guess at some sort of evil corporate motive or the involvement of stupid engineers, or even just dangling the implication that we've "Made a Final Decision" to store passwords this way, etc. It just isn't the case - our engineers are great and our motives are pure. If you want to blame anyone specifically, you can blame me for pushing the implementation in the direction I did. We're really just trying to do the right thing for our clients, and in this case, I took a great idea too far. Its just code, it can be changed - it will be changed, and changed in a way that will help our customer service staff continue to provide awesome customer service and also enhance the protection of our customers assets without stepping on toes in either regard.

We originally posted that commentary back in April because we had a ton of work backed up behind the release of our new domain and email management tools - which included a huge refactoring of most of the core code, transitioning to TDD and a ton of other important pieces. We were supposed to be done work on those pieces months ago, but as the management tools took shape, the task grew longer, pushing out these items to the point where it was getting embarrassing with our customers. That work shipped late last month and launches formally tomorrow putting us back in a place where we can get serious about the backlog. The new approach is pretty straightforward and moves us to a hashed password file, URL-based resets, etc. but also some identity verification features that our customers and customer support staff can use to validate who they are talking to in order to force resets manually. Its the validation piece that I'm most excited about given the extent to which the bad guys will go to phish a user out of their creds. We've seen some pretty sophisticated social engineering and we're hoping these new features will give our customers a leg up.

Sorry for the lengthy note - happy to take questions, slings, arrows, etc.

Ross Rader GM, Hover [e-mail]

[SkyMarshal]

Fwiw, let me share some of the less predictable consequences of what could happen if your pwd database is hacked, and why it's important to use bcrypt, PBKDF2, or scrypt to secure your users passwords. (http://codahale.com/how-to-safely-store-a-password/)

I was one of the folks whose email and password were compromised in the recent MtGox.com bitcoin exchange attack. Until then I had been using a three-tier password system, consisting of three passwords of increasing difficulty, used for sites of increasing levels of importance.

My bank/card accounts, email accounts, and any account that stored bank/card info (Paypal, Square) got the strongest password. Sites that were part of my online identity or similarly important, but did not store financial data, got the next strongest password (Twitter, Facebook). Finally, spam sites I didn't care about but needed a login for some reason got the third password.

Well, the MtGox hackers got my middle-tier password and the associated email address. Shortly afterward they also got my Twitter account, which used the same for login. Fortunately they didn't take the time to change my email address, and I was able to get it back with a password reset email.

And a few days later I tried to login to Amazon and found they had changed the password there too. I got it back the same way, pwd reset, logged into AWS and found my EC2 test instances had all been terminated and all the work I had been doing there gone.

Now I'm sitting here wondering what's next, as I can't remember all the sites I used that email/pwd combo on. But I'll never make that mistake again, and am now evaluating password managers like Lastpass, Keepass, Passpack, and Clipperz for storing unique, strong passwords for every site I use.

I'll also never use MtGox again, and have discovered a newfound wariness of all websites' security practices. One report like this is enough to make me not only file away the name of the site, but also the people who built it, as unreliable.

My point here is that, if your database gets pwned and distributed out to the black market, there's a realistic chance your users will be harmed in ways you haven't foreseen, on other sites not related to yours, and will remember and blame you for it indefinitely.

Given that most people have lots of sites they log into, and that most can't or won't remember separate passwords for them all, you can assume a good portion of your users reuses passwords.

The potential downside of those reused pwd's getting hacked via your site and put into the underground identity-theft rings and whatnot, far outweighs whatever user-experience upside you may perceive.

[ydm]

I agree with you wholeheartedly. Unfortunately, there will always be a few services that will store passwords in plain text.

Would unique email addresses for each service have helped your situation at all?

For example:

Facebook email: [e-mail] (forwards to your real email) Facebook password: password1

Hover email: [e-mail] (forwards to your real email) Hover password: password1

Bank email: [e-mail] (forwards to your real email) Bank password: password1

If any of those services get hacked (and the passwords are stored in plain text) then there's nothing connecting those accounts to each other since the email addresses are all different.

It's the system I use (along with 3 tiers of passwords not just 'password1' as used in the above example).

[s3b]

But then even if you remember the password, you'd still need to remember the right unique email id for each service.

[yuhong]

Not to mention it is security by obscurity.

[Flenser]

I had been using a three-tier password system, consisting of three passwords of increasing difficulty, used for sites of increasing levels of importance.

Now I'm sitting here wondering what's next, as I can't remember all the sites I used that email/pwd combo on

For my banking password I have a base password that I always add something to for each site in a way that I can remember without having to write something down. The idea being that although it might be obvious to any human looking at the password what I've done it's far more likely that attackers will be automating checking passwords on different sites and the program will just see my password failing on all other sites and ignore it.

[yuhong]

>Now I'm sitting here wondering what's next, as I can't remember all the sites I used that email/pwd combo on. LastPass lets you import your passwords from your browser.

[bluedanieru]

I also had my details leaks via the MtGox.com hack. Fortunately I have been using a password manager for years (Keepass) and don't share passwords site-to-site. So I don't fuck around with security, or try not to.

But, you do. It's hindsight, sure, but if you read HN you definitely know better, yet you did it anyway. You've learned your lesson, and hopefully the next time a service you frequent is hacked your exposure will be minimal. But it took something like this for that to happen. I'm thinking a lot of sites have had shitty security for years like Hover et al and are only now, with all the publicity surrounding recent breaches of security, beginning to realize they can't get away with it for much longer.

So just like anyone can cut you some slack, I can cut organizations some slack, for now, especially in cases like this where it looks like someone without the requisite technical expertise was given too much control over technical decisions (i.e. not the engineers' fault). That kind of shit happens all the time even if it ideally shouldn't. But, things have changed and security concerns have gained enough publicity that even clueless middle managers should have some inkling that it's important, so IMVHO if you haven't gotten your shit together security-wise as an organization by the end of this year, you're probably inept enough that I shouldn't be doing business with you.

In the meantime I'll practice the security diligence I preach.

[jat850]

Quoting you here:

"I'd also like to point out that the scope of the risk isn't trivial. For example, URL-based password resets are only as secure as the mailbox they are sent to. i.e. a significant number of domains are stolen and threatened to be stolen through email account exploits (re-registering previously used addresses, forwarding attacks, etc.) This is made even more complex when a domain expires and email on that domain stops functioning. Where should the password reset go? We get dozens and dozens of calls a day from people in this position that need our assistance, making it tough to simply send out a reset request."

Your point isn't invalid, but can be addressed in some way by a combination of a good customer support team, and two-factor or multi-factor authentication.

I think your transparency and accountability is great, but one thing I would say is this: This is a well-traveled discussion, no new ground is being addressed here. Your company has willingly made a tradeoff of security vs. usability. It's not one that I would make or accept (as a developer or a customer). But whatever solution your team may have chosen should be contrasted against (and I apologize for not having a real quote handy): Don't fool yourselves into thinking you can do security "better" than someone else.

As clever and intelligent as your team may be, time after time, weird encryption or password hashing or things that are homegrown implementations of these principles have shown to fail again and again and again.

So, knowing that, why would your team not opt for things that ARE vetted as being secure, trusted, open, and have widespread adoption?

Bridge the gap of customer frustration (and your other queries about domains disappearing, emails being lost) through good customer support. Otherwise it feels like you're ultimately only playing a game against time and inevitable loss/breach.

edited a few weird grammar issues

[freejack]

"why would your team not opt for things that ARE vetted as being secure, trusted, open, and have widespread adoption?"

It was a classic case of letting product management opinion over-ride engineering implications. Namely, on behalf of customer service, I went to bat - hard - with the engineers, to give our CSRs a completely effective way to handle inbound password requests in cases where customers no longer had access to their email account. I can't remember the exact conversation, but I could see the engineers at the time characterizing it as "being over-ruled". Long story short, brought forward almost two-years - we've got a new team on the project and I have a much greater appreciation of the subtleties and trade-offs and we've still got some work to do to fix my mistakes.

/r

[gnaritas]

I'd hope in the future, when your engineers throw a fit, you take some time to learn why you're wrong up front rather than overruling them. This was a huge mistake on your part and it's now public; you have a limited amount of time to fix it before you're hacked.

[freejack]

There was a bit more to it than that, but on this specific point, lesson well learned. I rushed this instead of looking for an agreement - as they say, measure twice, cut once.

[GigabyteCoin]

What's to stop me (possible hover CSR, co-worker, whatever) from simply reading a high-profile customer's password and transferring all their million dollar domain names to me later that night when I get home from work?

I have a pretty good memory. I bet I could remember 5-10 simple passwords and email addresses without writing anything down. Chances are the idiots use the same password for their email anyways.

Muahahaha, I'm rich, and your company is going down the tubes in a lawsuit. See you later!

[freejack]

Industry secret - at the most basic level, registrar and registry staffers don't need access to a customer account to manipulate a domain name. We've implemented tons of controls to manage who can do what, etc. but relying on customer passwords to safeguard domain names from internal tampering isn't really a great tactic.

[pavel_lishin]

What about the same scenario, but instead of altering domain records, a CSR logs into the customer's e-mail account, or bank account, and starts wreaking havoc?

[Dylan16807]

What? If it's the same scenario then the CSR does not and never did have the password, they just have domain control panels. The whole point is that they can't do that.

[keeperofdakeys]

"...to give our CSRs a completely effective way to handle inbound password requests in cases where customers no longer had access to their email account."

I personally don't see how having the plain-text passwords help in the case where the person owning the account doesn't have access to their email account. Since they don't have access, you can't exactly email them their password.

[freejack]

Most of our customer inquiries come in over the phone.

[JonoW]

Assuming you verify users over the phone securely, why not just reset their passwords for them with a one-time use, temporary password? Surely that's not much harder for the end-user to deal with, and then you needn't store their password...

[bluedanieru]

Hover customer here. Please don't do this again.

[freejack]

Absolutely.

[drgath]

Broadcasting this fact is almost as bad of an idea as implementing it in the first place. You now have a bullseye on your site from every blackhat reading this. It's your company and your customers (which I am not one of), so most of the time I would just say do whatever you want, because it doesn't affect me. Problem is, it does. It affects every developer out there, because once your security is compromised and every password is leaked by LulzSec, Anonymous, ScriptKiddies, etc... we're all at risk.

I understand you did this for your users and your product, but please reconsider. Your users are also everyone elses users, so your lack of proper security is shared amongst all of us.

[8ig8]

Thanks for taking the time to respond. At a minimum, it's good to know that you're a HN reader.

I've got 100+ business domains at GoDaddy. I've had a todo to move these somewhere for a while now and Hover is (was?) my top target. Coincidently, about a week ago I opened a Hover account and registered my first domain there.

Hover got my attention because you seemed to be the anti-GoDaddy. Now I may re-evaluate my options. Hopefully security may get a re-look from you guys by the time I get to that pesky todo on my list.

BTW: I'd love some kind of corporate level account for customers with my level of domains. I'm small potatoes for some of the other corporate domain registrars. Thanks.

[freejack]

Thanks for the note. We're lining up to knock this out quickly. Drop me a note to [e-mail] if you like and i can fire you an update when we update password security and the rest of the related tools. We can set you up with a corporate level account of course, just let me know if/when you are ready.

[8ig8]

Thanks, Ross. I'll drop you a line. I appreciate the offer.

[ohashi]

With 100 domains you may want to try contacting Fabulous.com. They live up to their name. You're a bit low on the required names but worth a shot. If you get in, you'll be saving a lot more money and the service is the best in the business.

[8ig8]

Thanks. I took a look at Fabulous.com. I recall checking them out once before based on a HN recommendation. They're out of my league. When I say I have 100+ domains, I mean like around 105 (give or take). I'm not really a 'domain professional.'

For the reference:

  > To be eligible for a Fabulous account we
  > require that you meet at least one of the
  > following:
  > Domain portfolio must generate US$750+ per month
  > Transfer 750+ domains to Fabulous

Source: https://secure.fabulous.com/signup.htm?formcode[event]=signu...

Edit: Fixed my wacky quote formatting.

[ohashi]

I know what they quote on their page. If you're really interested contact me from my profile and I will see if I can help you out.

[8ig8]

Thanks, Kevin. I'll take another look at Fabulous. I quickly wrote them off.

(Man, I wish I owned the Fabulous.com domain. My business cards would be bedazzled.)

[balloot]

A couple things here:

1) I simply don't believe your claim that the number of stolen emails is so high that sending password retrieval links via email is unfeasible. This isn't a new problem that is just faced by hover.com, and most solve it without resorting to plain text passwords.

2) You aren't taking a wide enough view here. By storing and sending plaintext passwords you are doing more than making someone's hover.com account insecure. You are creating a weak link that may reveal a user's password that is used in any number of places. If I were to use hover.com, and someone got access to your DB, they would get my 2nd tier PW and instantly have access to my Facebook account and a number of other things that could cause havoc in my life. This seems obvious, but there is little to no acknowledgement of this fact in the post above.

3) Trust has been lost, as account security is clearly not a priority among your engineers. You may change things, but those changes will be made by someone who thought storing plain text passwords and sending those passwords in email is OK. Even after passwords are no longer sent around via email, who knows what kinds of other security flaws will remain that aren't clearly explained on the corporate blog?

With all this being said, it's great that you came here to acknowledge fault. Best of luck to you, but I have to say that I won't ever be a customer.

[freejack]

I don't disagree with most of what you are saying, and would add that good security practice is a shared responsibility. Sharing passwords across sites is the user equivalent of storing passwords in plain text. i.e. It might be secure enough until its not.