[tscanausa]

My name is Terrance and I work on Google's Cloud Support Team. Our team mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's experience obviously shows that we fell short this time. The process should have been an easy flow to follow, and it was not. We are reviewing this incident in detail to ensure that we make the process less error prone and quicker in the future.

Best, Terrance

[ufmace]

Thanks for your help and for responding here, but I hope you guys at Google know that this seems like a major organization-level problem. There's a big perception that Google has no real customer support for anything, even if there's pretty serious amounts of money involved, fueled by a regular drumbeat of blog posts about people who had their entire livelihood shut down for unexplained reasons with no possible recourse. It's kind of like the "five whys" failure analysis. Yes, this particular automated workflow was broken, but that's not the real problem. The real problem is the lack of human support that can turn issues like this from a crisis to a minor hiccup.

It seems like you guys really need to get some call centers filled with people with some level of diagnostic ability and problem-solving authority. Yes, this is expensive and difficult to get built, but if you really want to Reduce Customer Anxiety, nothing can do that like the knowledge that there's a real person available 24/7 who can fix any problems on Google's side. Make it pay per incident or by subscription if needed to keep the jokers out, but anything is a whole lot better than nothing.

[bigiain]

Even the phrase "Reduce Customer Anxiety" would increase my anxiety if I were a customer (and steels my resolve not to become one of their "acceptably-anxious customers").

"Reduce", not "remove" - it's stinks of implying there's an acceptable level of "customer anxiety" that their platform should generate, and someone's set some easily-measurable metrics and a bunch of people's KPIs - and those people are now all working out how to game the metrics instead of solve the problems. "Hey, my bonus depends on reducing the clicks on the "Request an appeal" button. What's the easiest way to meet that goal? I _could_ start handling customer service in a professional and sympathetic way - but that's not "The Google Way". I know, I'll just move the button to a different place this quarter, rename it next quarter, and remove it all together before the end of the year."

Sorry Google-people - I know you all don't think you're doing evil, but there'a a hell of a reputation you got to overcome on the "Google just doesn't do customer service" issue.

[ufmace]

> "Hey, my bonus depends on reducing the clicks on the "Request an appeal" button..."

That sounds entirely too plausible as the real root cause of this problem.

[1_2__3]

I don't want to sound disrespectful, and very much appreciate your posting. But realize that the audience here has seen the OP's story play out with Google's products over and over again. This is clearly a systemic issue with how Google handles support, and claiming "fell short this time" is more than a little disingenuous.

(I also get 'man bites dog' will also get more traction than the inverse, but regardless - this kind of complaint is a very common one about Google's products, from consumers and businesses alike).

[Lewisham]

FWIW Google Cloud Support != Google support in general. My interactions with Cloud Support on the other side of the fence has led me to believe they are very committed and working very hard. Yes, you have to pay for it. But once you do, regardless of the tier of support, I've seen them work tirelessly and escalate tickets to engineering quickly (order hours, not days) if they couldn't figure it out.

I'm looking at the postmortem now and without wishing to jump the gun and talk about things I can't, it looks like this is being taken very seriously and a number of improvements and bug fixes are going to result. In this instance, I think it's doing the Cloud Support people a disservice to call them "disingenuous".

Disclaimer: Used to work on Google Cloud, now on Google Open Source Programs Office.

[spacehunt]

> FWIW Google Cloud Support != Google support in general

> I think it's doing the Cloud Support people a disservice to call them "disingenuous"

The problem is that to us from the outside, they do look the same.

In fact, that email in the article, and the subsequent events, are in exactly the same style as the "ban hammer" Play Store publishers get. You really can't blame us (Google's actual paying customers) from equating one with the other.

[Lewisham]

I'm not entirely convinced that people who have a bad experience with Windows support would then assume Azure support was poor, similarly with Amazon and AWS... Why should Google [Product Area X] and Google Cloud be any different?

[spacehunt]

Windows and Amazon are consumer facing products. Azure, AWS, Play Store publishing and Google Cloud are all developer facing products.

(And no, I do not equate consumer Windows support with enterprise Windows support either, and as it turns out they are not the same level.)

[x0x0]

Nobody cares how hard you or your coworkers work. Results matter, nothing else.

Given the story so far, you offer adwords style customer service (we don't give a damn about you) while attempting to catch up from position 3 behind aws and azure in the paas race. Complete with automated emails with outdated documentation and no human anywhere without a social media outcry. I'm sure stories like this are damaging (seriously, just use aws), and there will be some postmortem damage control, but unless I see a human support sla for situations like this, I can't see why anyone would risk their business on you.

Seriously, a fix it or we shut you down email for reasons you won't share with an 2 business day response. Of course, you're issuing a 3 day warning, so if that fell on a Friday, you would be dark before anyone bothered to look at the support ticket (3 days expire Monday, 2 biz days Tuesday evening). Amazing.

[andmarios]

As someone who had a similar experience[1] with Google Cloud, I would like you to understand that in Google Cloud people place their businesses. When you switch off a project, you switch off someone's business. For anyone else, even for a goverment agency, such action would never be taken hastily. Since downtime could be catastrophic for a company, you should err on its side. Technical prowess isn't all that matters for business.

That said, we still use Google Cloud for some part of our infrastructure and enjoy its technical side.

[1] The billing credit card for one of our projects expired weeks before any invoicing would happen. Instead of informing us —after all we were paying customers for many months before, there should be at least some good faith— all our projects were disabled many minutes (maybe 15 or 30) before we got a confusing email about the issue at hand and the total downtime was two days.

[Animats]

Love the "Reduce Customer Anxiety" phrase.

Reminds me of the emergency telephone signs on the Golden Gate Bridge. They once read "Emergency Telephone in case of Breakdown", so people could call a tow truck. Then they were changed to "Emergency Telephone for Psychological Counseling" and now connect to suicide prevention. I can see someone with a flat tire on the phone, "And how does having a flat tire make you feel?"

This mindset is totally inappropriate for a B2B service. The caller is probably not anxious. They're at work, doing their job, keeping something working. They need repairs, not grief counseling.

[1_2__3]

It's the new fad of marketing support as being an advocate for the customer. They use language and phrasing to imply they exist to make the customer happy/relaxed/productive which if it were true would be a great thing.

But it's just marketing, at the end of the day support is still a large expense that some companies believe they don't have to bear. This seems to work for some companies better than others, likely due to different business models leading to different customer expectations and requirements.

[noja]

Don't worry dear customer, the company "understands your concerns". Great! (But that doesn't hel me get my problem solved.)

[aluminussoma]

Google Cloud publishes a Root Cause analysis after technical incidents. I wonder if this non-technical incident is worthy of a root cause analysis. As a potential customer, I know I'd feel more comfortable seeing this organizational "bug" fixed.

[jschwartzi]

It seems like there are a lot of points in Fred's story where even a miniscule amount of manual testing would have revealed the problem. For example, paying a tester to follow the instructions in the FAQ or to try to appropriately respond to the original email would have revealed how unclear the instructions were and that there is a missing UI control.

[spacehunt]

I really want to believe you, but every Google service has been like this for years. Like Play Store for example.

It will take way more to show you guys really are reviewing similar incidents and, more importantly, to actually act on your reviews and improve your processes.

[bitmapbrother]

This is absolute BS. I've had nothing but stellar support from the Play store and you can easily contact them via chat, email or even voice.

[lowleveldrone]

I just tried publishing my first app to the Play store and am in the middle of a horrible experience.

I wrote a little app to help myself learn birdsong. You copy some mp3s to a folder on your sdcard, open the folder in the app, and it will shuffle them and only show you the name of the file if you ask. Pretty simple, but I found it useful and thought it might be useful to other people. So I pay my $25, upload my app, and wake up the next day to an email saying my app had been suspended for "deceitful behaviour" and that it happened again they might suspend my other Google services, for instance the gmail account I've had since 2004.

Of course the email doesn't contain any information on what the deceitful behavior was, so like a Kafka story, I'm stuck defending myself against charges that I'm never informed of. When I heard back from support, they say that my app opens to a list of mp3 apps to download. So either my account has been hacked and someone changed the app apk, or they are unable to tell the difference between a directory listing from the sdcard and a list of mp3 apps. If the latter, how my app can be deceitful when it matches one of the screenshots I attached is completely beyond me.

Once my email migration to fastmail is complete, I'll rename my app and try again, this time with a modified UI that makes you click a button before showing a directory listing. I'd like to get my $25 worth if nothing else.

This whole incident has left me with zero confidence in their customer service or technical competence.

[hga]

It's this linkage with the rest of your accounts that can make doing business with them particularly dangerous. After the Google+ real names debacle started, which included people using their real names which Google didn't believe, and shutting down all their Google services, I stopped using every one of their services except for ephemeral zero stickiness ones like search (fortunately I'd discovered Fastmail some time previously, and they're beyond wonderful).

Buying a house built in 1910 and having to remodel and move to it financially prompted me to buy a smart phone, and Project Fi was clearly the best match, so I've now seriously reversed that ... but that also means I now won't even think about increasing my exposure to these sorts of risks by trying to write apps for the Play store, using any of their cloud offerings, etc. My main phone and its number are too valuable, too sticky to risk.

[chinhodado]

Just because you have not experienced it doesn't mean it is BS. There are countless horror stories out there about people getting their apps removed by an automated process, receiving templated emails with ambiguous reasons for the removal that is impossible to pinpoint the exact cause, and having no way to contact any real person to ask about it.

[frakkingcylons]

Please, please post an update on what y'all are going to change in the future to prevent incidents like this from happening again. It would be more reassuring for those of us who are considering buying into Google Cloud Platform.

[forgottenpass]

This post sounds like you copy-pasted it from the your department's list of prewritten generic reassuring statements. I doubt that is helping the perception of Google's support.

[eadz]

How does one reach someone at google in an emergency when you have no access* to any google services?

* locked out of your account

[nikanj]

Write a good blog post and get it to the front page of a news aggregator

[tscanausa]

We became aware of Fred's problem before we knew it was here.

-Terrance

[CountHackulus]

That doesn't really answer the issue, or the question. How are we supposed to get in touch when things go wrong? The blog post clearly showed that support was an afterthought and not properly QA'd.

[jjjjoe]

Post to one of the product-related Google Groups like google-appengine@ or [e-mail]. We have support staff monitoring those lists seven days a week.

[arkitaip]

You know that Google has a serious problem with handling complaints and feedback when people take every opportunity to interact with an actual Google representative no matter what the topic is or if they are indeed interacting with a real Google employee.

[ben_jones]

Personally I really value comments like this and therefore feel the need to point out that the statement "Our team mission is to Reduce Customer Anxiety." seems like a bullet point for a status meeting and likely does not convey the desired effect of such a post: to have a somewhat personal response from a normally faceless company.

[CaptSpify]

It actually gives me the opposite feeling than probably intended: "We'd rather pay someone to post BS responses on social news sites, than dedicate someone to fix the problem."

I'm not saying that's actually what is happening, but it feels like it to me.

[blakeyrat]

If people wanted their anxiety reduced, they'd sign up for a guided meditation group.

People want their accounts reactivated.

[x0x0]

So what's the support process when you screw someone but they can't hit the front page of a social media site?

[jjjjoe]

Assuming that the appeal process is broken, or you're locked out of your account or something equally egregious: you post to one of the product-related Google Groups listed at the bottom of cloud.google.com/support. Support staff and engineers read those lists and will escalate issues.

[tscanausa]

HN is limiting my post rate. So I can't respond to all of you. If anyone would like to chat about reducing customer anxiety or feedback on what you would like to see on this process, you can send me an email and I will respond when I can.

Tscanausa @ Google

[breakingcups]

I think the response has been pretty clear and near-unanimous. Here's a bullet-point version:

1. It's absurd that an automated process accuses you of trying to intrusions of third parties without providing details of what has been detected. This leaves the "accused party" (your paying customer) without ANY idea what is wrong or how to fix it.

2. The automatic suspension of the entire project within 3 days is similarly out or proportion, especially without Google attempting actual human contact first (a phone call, for example).

3. The appeal-process is apparently literally broken. Even if this weren't the case, disabling a customers entire project in an automated fashion requires WAY more ways of directly reaching Google to prevent this. Give a direct phone number in "emergencies" like this. We know it costs money and training. We expect you to accept this as a cost of doing business.

The above leads to the (at the moment rightful) impression that Google Cloud isn't ready for business, especially not small business. Turning off what could be someone's livelihood is a testament to the (possibly unintentional) arrogance that is widely perceived from Google.

[ufmace]

So you're telling us that you're frustrated because an automated system prevents you from doing something you need to do, and there's nobody to appeal to...

[abrookewood]

You need a bigger team.

[lima]

Please offer an option to host servers in Germany under German data protection laws like Amazon does.

There's very little information and/or reassurances.

[snowwindwaves]

I think tscanusa might be trolling HN