[matthewdrussell]

Disclaimer: I'm CIO @ Namecheap

1. The credentials were resent to an already compromised email account

2. This is an isolated case

3. Established procedure was not followed

4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved

5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups

[swanson]

My hobby: role-playing how I would respond as the CEO if my company was getting skewered on HN. Here is my version!

---

Disclaimer: I'm [not] CIO @ Namecheap

We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.

In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.

We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>

I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.

[tamar]

^^ we'll take it. We've been responding for the last 1+ hour to things in real time across several social networks, so we're a little rushed. But thanks for the role play :)

[dennisgorelik]

I like original matthewdrussell's version more.

Overblown/fake apology is not very informative - it's hard to say what exactly can you trust in it.

[helloguille]

You are really good at it

[biot]

Previously, IFTTT: https://news.ycombinator.com/item?id=11379475

[SolarNet]

If I ever run a company that screws up I'm calling you.

[alasano]

I like your response! If only for the fact that I'm seeing people taking apart the real CIOs response like it's code because it's in a numbered list.

[kelukelugames]

That's impressive, can you teach me to write like you?

[swanson]

Formula:

* Actually apologize in a human way

* Show empathy by identifying the impact of what happened to customers (not your impact internally)

* State action items that you've created, even if they are just in 'evaluation' state

* Indicate that the specific incident in question is being handled outside of this forum

* Take responsibility for things even if you shouldn't "have to"

[voltagex_]

For a "what not to do", have a look how (the CEO of?) FTDI responded after they were caught intentionally "bricking" chips that were detected as counterfeit by the Windows drivers.

[Namidairo]

Last I heard, these tainted drivers from FTDI weasled their way through WHQL and into Windows Update...

[kobayashi]

If you're actually interested in the topic, here's an absolutely fantastic blog post on the subject:

http://blog.statuspage.io/why-public-apologies-suck

Some important bits:

>4 PARTS OF A BAD APOLOGY

- Justifying the offending actions or words.

- Blaming the victim.

- Making excuses.

- Minimizing the consequences.

>8 PARTS OF AN EFFECTIVE APOLOGY

- You actually have to use the words I’m sorry.

- Acknowledge that you messed up. (As in, “I take full responsibility for my words.”)

- Tell the person how you’ll fix the situation.

- Describe what happened, but without foisting the blame off on someone else.

- Promise to behave better next time.

- Make sure the person knows you know exactly how you hurt or inconvenienced them.

- Much like the first rule, it’s important to use some version of the phrase “I was wrong.”

- Ask for forgiveness.

[infinite8s]

It just requires thinking from the perspective of the person reading it, instead of trying to CYA.

[mightybyte]

Go out right now and get the book "Crucial Conversations". It is BY FAR the best book I've ever read on this kind of thing. It is simultaneously the best relationship book I've ever read and the best business book I've ever read. It goes through the basic principles for handling these situations in an easy to understand way.

[kelukelugames]

This should become a thing. I want to make a Tumblr now.

[snowwolf]

> 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved

This is not the correct solution. What's to prevent the next new person from making the same mistake?

If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.

People will always be fallible.

[fencepost]

This is the kind of 'learning experience' that becomes part of corporate culture and future training. When someone says 'Why bother with all this?' the response can now be 'Read this writeup of how ONE person NOT doing this correctly cost the company a ton of marketing $$$ and STILL left us with a black eye with our more technically-savvy customers.'

And how many people at Namecheap do you think aren't aware of this by now?

But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).

[tamar]

That's part of the whole issue. It wasn't simply an issue of retraining. The entire company is well aware of this issue and is using it to improve, not simply to reprimand a single person.

[vehementi]

Putting #1 as #1 looks like bitter deflection. You do it elsewhere in the thread too, saying that lack of 2fa on the email account opened the door to this. You should be well aware both that most security issues end up being perfect storm of circumstances, and that attackers can and will target multiple points in the chain. Relying on #1 as the spearhead of your apparent defense here is tantamount to admitting that you are relying on the security of people's email accounts as part of your own security process, which is wild.

You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.

[rudolf0]

It is petty to list it as #1.

However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.

This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.

[coalescence]

With #3 - ideally your systems should not allow you to break established procedure. Mitigate the risk by not giving the support staff tools to shoot yourself in the foot so easily. This could be achieved with peer verification or some other mechanism (lots of ways if you think it through).

[tamar]

Yes, learning experiences are had when things happen like this. We look to the future, not to the past, to ensure the same mistakes do not recur.

[chinathrow]

3. Established procedure was not followed

Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".

In this case, the support staff wouldn't even needed to be trusted in the first case.

[vblord]

This is a great point. The software should be modified to not allow the employee to even make any modifications to the account without the correct credentials.

[matthewdrussell]

Agreed, and we're looking to improve this area too.

[Khao]

> Established procedure was not followed

Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.

[SeanDav]

People make mistakes. The customer support person was probably just trying to be helpful and not fully aware of all the ramifications. This is unfortunate but presumably there has been some retraining.

Note: I have no direct or indirect relationship with Namecheap at all.

[teraflop]

Sure, but retraining doesn't change the fact that people make mistakes, so it doesn't really solve the problem at all.

[tamar]

Sure it does. People in positions like this are required to learn. If they repeat serious mistakes like this, they're not the right fit for the job.

[eximius]

Hopefully because the majority will follow it? We have no statistics on this, just this one case that failed.

[tamar]

Khao - the matter was addressed and the staff was retrained to close the gaps in procedure.

[sneak]

If 3 is possible, how are we to believe 2?

[tamar]

I guess take Matt up on his offer where he said this: "Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place."

[jc4p]

I love this. Am I wrong or is it "if you don't believe me, try the social engineering hack yourself!"

[kelukelugames]

I love namecheap but 5 sounds like victim blaming. Come on.

EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.

EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.

EDIT3: congrats to Tamar for being promoted to a Namecheap executive!

[coalescence]

Low end hosting doesn't generally have backups, because it's well, cheap. Extra overheads make the price increase, then you're not cheap and can't compete at that end.

Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.

[drostie]

Furthermore, is Namecheap authorized to copy their clients' data by their terms of service? If not, automatic backups may bypass totally-reasonable expectations that other users have. Backups can potentially be a threat vector, for example. There might be many reasons why one of Namecheap's clients might say "you copied this data?! and now I have no control of the environment the backup lives in?!"...

An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.

[coalescence]

That's true, although we should be fairly concerned about a company using very cheap hosting on VPS with no form of encryption storing anything sensitive. You may also be breaching PCI DSS (fwiw) doing that.

In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.

Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.

[ChrisDutrow]

> Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.

They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.

Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.

[coalescence]

There are a couple of adages that may be valid.

"If it's too good to be true, it probably is"

and

"Cheap, Good, Fast - pick two"

[matthewdrussell]

Everyone should practice a good backup routine and take responsibility for backups.

[kelukelugames]

This is not good damage control/PR. You are letting ego get in the way.

[matthewdrussell]

I respectfully disagree. I'm here, along with Tamar, reviewing and considering each point posted. There's some good suggestions and we're listening.

The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.

[peterwwillis]

Imagine you just lost two servers you can't replace, or you're a potential customer reading this thread, and are afraid of the same.

This is what they read as the company's response to this loss:

"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."

Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?

Now consider this reinterpretation of the statement:

"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."

[beeboop]

I am not sure why you would respond to an accusation about victim blaming by reiterating the exact thing that caused the accusation. You might want to reconsider continuing this particular aspect of discussion for PR reasons. It's not an argument you're going to win.

[mindcrime]

It's not an argument you're going to win.

Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?

[beeboop]

The issue is that Namecheap was the one that fucked up here, and now is not the time to emphasize "you should really be prepared for us fucking up in this manner". It's victim blaming. It looks shitty. The argument I refer to isn't "you should have offsite backups". The argument is that Namecheap is implicitly victim blaming, and they're not going to convince many people that they aren't.

[mindcrime]

Eh.. I don't really agree that this is victim blaming. But then again, I find that I disagree with most uses of the phrase "victim blaming". Pointing out that somebody did something sub-optimal, while still acknowledging the mis-deeds, mistakes, etc. of other parties, is not "victim blaming" in my book. It's just pointing out the truth.

I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.

[tamar]

It's not victim blaming. It's simply a reiteration that it helps to have this in place if you are specifically opting to rent/lease a server that does not offer it.

Also, it's stated in the knowledgebase that it is advisable to set up server backups of your own if you do not have a managed server: https://www.namecheap.com/support/knowledgebase/article.aspx...

[TheSockStealer]

Agreed. However, is this messaged anywhere in your documentation or setup instructions? Do you provide instructions how how to set this up with a 3rd party or list of 3rd parties?

Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.

[tamar]

Yep, it's said here: https://www.namecheap.com/support/knowledgebase/article.aspx...

[endemic]

When the product is an unmanaged VPS, I think certain assumptions can (hopefully) be made about the capabilities of the customer.

[tamar]

I don't think it was personal, simply a reminder that it always helps to have good backup procedures in place.

Even my managed services have offsite backups. Better be safe than sorry, I always say.

[kelukelugames]

"Better be safe than sorry" - namecheap for when you lose your stuff on their services.

I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.

[tamar]

That comment I made refers to data integrity across platforms. You should be smart about data, no matter where it arises, if it is important to you.

For example, let me give you a look at what my Windows hard drive looks like.

My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.

Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.

So yeah, I repeat, better to be safe than sorry. Your mileage may vary.

[HemanHeartYou]

"Hard drives never fail" - kelukelugames

[dang]

We've banned this account for repeatedly violating the HN guidelines. If you don't want it to be banned, you're welcome to email [e-mail].

[tamar]

Doh.

[tamar]

Re: Your edit - just a note, it is very clear here that in points 1-4, Namecheap has acknowledged a mistake. That's exactly why there was a lot of training (and retraining) internally to ensure this mistake does not recur. But we do acknowledge it is an isolated incident. That doesn't mean it's not less important - we're fully aware of what happened here and it will not recur.

[tamar]

Thanks for edit2 :) I see us having used the word "mistake" many times here! But yes, we apologize that this happened as well.

[kelukelugames]

The namecheap CIO never uses the word mistake. The execs rarely show any remorse. Best case they delegate to underlings like the social media guru.

[tamar]

Let's not throw personal attacks at me (and the tongue-in-cheek "congrats for being promoted to executive!" comment). There's plenty of remorse and there's plenty of acknowledgment of mistakes here. That said, as we acknowledged elsewhere, we're responding to the matter across several different platforms and specifically say we're rushed in trying to get out some basic insights behind what happened and transpired. A more well crafted blog response for all to see (this time from the CEO) has been published to https://blog.namecheap.com/social-engineering-issue/

[kelukelugames]

It's a common practice. I don't see how it is personally offensive to you. My description was for the CIO and execs in general, yet you insisted they were for you. So maybe you should stop making tongue-in-cheek comments. In fact, I am moving my domains off of Namecheap because I don't think Namecheap is very good at handling customer relations, particularly on social media.

[tamar]

(the reply link didn't show up before, so I don't know if you saw my response posted right after this)

To be fair, your third edit was only for me. And that was the only comment I was replying to.

As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).

It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.

[Patriotspade]

Welcome to the jungle. Kumbaya.

[mtgx]

Regardless, to fix this PR disaster, I suggest you add some strong and perhaps just as importantly modern security features in the future that would regain you good will with HN types (and therefore everyone else).

And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.

[opendomain]

Matthew,

It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures.

I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations.

OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.

[tamar]

OpenDomain -

We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.

[alainv]

The offer of a free year of hosting is nonetheless a paltry joke. The high road here is to acknowledge that customer may choose never to host with you again and still go above and beyond in attempting to make it right to them, e.g. by offering a full refund for the last year of hosting they'd paid for or the like.

[tobltobs]

Tell me one registrar where you can be sure that this will not happen and I will move my domains today. I wouldn't even care if I have to pay 100$ per year for a domain.

[frankchn]

MarkMonitor perhaps? Google and Facebook both use MarkMonitor as the registrar for their primary domains. Might be more than $100 per domain though.

[mgo]

Try tens of thousands.