[andr]

I have no relation to DO, but I'm surprised by the negative responses in this thread. I can't think of any other major company conducting a public postmortem for a customer service failure (as opposed to networking/ops failure). Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!

And of course DO will still retain the ability to suspend your account for suspected fraud - that is the case with any cloud services company, and any online business in general (check your ToS). Again, I can't think of any business that will en masse promise to never react to any fraudulent users. It's how this process is performed that matters and that's what they are improving.

[dang]

> I'm surprised by the negative responses in this thread.

That is actually a general pattern. Negative, dismissive responses come fast because they're reflexive. They don't require processing significant information, nor reflection, nor thoughtful writing—all of which take time. Therefore they are the first to appear.

After that, a second wave of objections gets triggered because people read the first wave and are dismayed by how negative it was. That is the contrarian dynamic: https://hn.algolia.com/?query=%22contrarian%20dynamic%22&sor....

[thaumaturgy]

Yep. I haven't had the best experience with DO previously (not terrible, just not great), and was vocal in the original thread about this, and I think this is one of the best reactions to a customer service failure that I've seen from a tech company.

This wasn't a failure that impacted thousands of customers [#]. DO could've just fixed things up for this customer and said not a word more, and changed nothing, and everyone would've forgotten about it by about ... now.

Instead they dedicated a nontrivial amount of resources to understanding what went wrong -- identifying not just a single cause, but several -- and publicly explained what happened, without a lot of weasel words, and what they're going to do to fix it.

It's an awesome response.

[#]: ...at that specific time. Yes, others have probably previously been impacted.

[twblalock]

This happens every time a post-mortem is posted on HN. It is often dismissed as a PR move, or ass-covering, or lies.

[rlue]

I haven't been around the block enough to know, but I'll take your word for it.

What I don't understand is why anyone would complain about a post-mortem as PR. If that's what DO was up to here, I'm sold—it shows transparency, thoughtful problem analysis, and swift execution.

More than what this means to me as a customer, I could surely stand to learn a lot in my own work from the way they approached this situation.

[dkersten]

A PR move isn't worth anything unless they actually follow through. I suppose many people are skeptical that things will actually improve as PR is often all talk and no actual action. Not saying that's the case here, but it certainly is difficult to trust a company purely by what they say for PR these days.

[microcolonel]

There's no amount of money you can pay them to get decent support, that I'm aware; at least, if you could, you can't just pay them upfront. They screwed up their weird bespoke network configuration system (which they inexplicably use in place of DHCP) on FreeBSD, and their only support output is: lol blow down your instance and start a fresh one.

You tell me why a company like theirs, which should be mature, ditched their proprietary support and ticketing system (which actually worked) for an shoddy, misconfigured off-the-shelf product which has equally little excuse for being that bad.

I think it's customary here to be unfair to companies, even when they're doing "the right thing" as DO is right now; but DO has spent their customer patience budget elsewhere, so I'm not going to be surprised if people view the post mortem as an inadequate replacement for getting it right the first time, rather than a followup to an honest mistake they will actually try hard to avoid in the future.

[julianlam]

> There's no amount of money you can pay them to get decent support, that I'm aware;

We at NodeBB have a high enough spend to have access to level 2 agents (their responses are around 1 hour turnaround, if not sooner).

We don't actually spend that much either, compared to what some of our clients pay AWS, etc...

Now, we certainly don't qualify for their highest tier with the dedicated support manager and slack access, but that's ok. DO's been amazing to us as a host.

[microcolonel]

Yeah, I'd just like to be able to pay for the support, without having so much volume. I can't really consider sending them enough business that we'd qualify for level 2 agents if I run into regular technical issues with the services before I even get that chance.

Is it bad for image if they just have a "send a couple hundred bucks because you need to speak with someone real bad" button?

[codegeek]

What is the high enough spend average to get Level 2 agents if you don't mind sharing ?

[HappyDreamer]

I wonder this me too :-)

Their pricing page reads: "world-class technical support to all of our customers—around the clock" (https://www.digitalocean.com/pricing/#Included_services ) b.t.w. which doesn't seem totally accurate to me, with the 12h and 29h response times in this account banned case.

[julianlam]

We qualify for their "Business Support" tier: https://www.digitalocean.com/support/#BusinessSupport

[julianlam]

Hey, sorry for the delay -- the info sheet they provided us lists $500 monthly spend.

https://pages.news.digitalocean.com/n/b0000VI6mEn0DfeZ0305PT...

[_Codemonkeyism]

DO has been amazing for us too.

[pbreit]

I don’t understand how the account did not have payment history. Hadn’t it been in operation for quite some time?

[rajaganesh87]

I think the account used startup credit.

https://www.digitalocean.com/hatch/

[manojlds]

The tweet thread suggested using for years. So either they are giving credits long time for startups they don't even know how and what they are doing or messed up a simple logic about history of the customer.

[dosy]

Similarly, as I have skin in the game and didn't want to get blocked by DO, I was expecting a muddy response and ready to make a throwaway account and complain about "keeping processes opaque to give carte blanche to take whatever arbitrary action they like" in the normal complaint about deplatforming and tech censorship, but then I read their document and was surprised and encouraged by how transparent DO were. Congratulations DO!

[computerex]

> I can't think of any other major company conducting a public postmortem for a customer service failure (as opposed to networking/ops failure).

There are many companies that have done this in the past. They are not doing this out of the goodness of their hearts, this is lip service for the fact that their mishap blew up in their faces on twitter. Do you really think they would have gone at length to highlight to the public this incident had it not gone viral?

> Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!

There is no telling that they are actually going to follow through with anything. Mere lip service.

The bottom line is, people host their businesses and livelihoods on cloud providers and they (the cloud providers) should take the necessary care and precautions when taking destructive actions. Maybe err on the side of the customer instead of shutting down someone's entire business because of some automated heuristic. Maybe have a better response time than 29 hours. Maybe teach basic communication and develop processes so that care agents can see and react appropriately to recent activity on the account. These are not revolutionary concepts, they are simple things that demonstrate customer care, something DigitalOcean is sorely lacking.

[rajaganesh87]

> precautions when taking destructive actions.

No data was lost, it is not destructive in anyway.

> because of some automated heuristic.

If the customer had "payment history" none of this would have happened. Probably it was being used under "startup credits"

> people host their businesses and livelihoods on cloud providers

people shouldn't run entire operation on credits and blame DO in twitter.

Only issue is that DO took 29 hours, apart from that i see no problem with DO.

[icebraining]

> people shouldn't run entire operation on credits

Why not? Until now, I wouldn't have considered that using credits might make me a second-class customer. They should at least be upfront about that.

[dkersten]

> No data was lost, it is not destructive in anyway.

Except in the way that the guy may[1] have lost customers or revenue due to the downtime. Being offline, even without data loss, is very destructive for many businesses.

[1] I don't know anything about his business.

[computerex]

> No data was lost, it is not destructive in anyway.

Tell that to the owner who was begging DO for their data back on Twitter. Again, had this not blown up on twitter nothing would have been done.

> If the customer had "payment history" none of this would have happened. Probably it was being used under "startup credits"

What's your point in saying this?? The fact is that the customer faced downtime because of a bug in DO's code.

> people shouldn't run entire operation on credits and blame DO in twitter.

Are you saying that customers on credits aren't subject to SLA's?

> Only issue is that DO took 29 hours, apart from that i see no problem with DO.

I think you seem very biased.

[pbreit]

It should be pretty hard to shit down a legit biz. Seemed automatic in this case.

[gavindean90]

I mean it is. Unless your business is wholly dependent on a service from my business.

[pbreit]

Edit: shut

[scarejunba]

What? We use a few million dollars in GCP credits every month.

[_Codemonkeyism]

"There are many companies that have done this in the past."

Haven't seen those, can you point me to them? I love companies doing this.

[artificialLimbs]

There is also no telling if they're not going to follow through, and it is not mere lip service.

What cloud business do you run that does better, according to your standards?