[tcdent]

The interesting and yet-to-be-explained part is why google flagged the account?

Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause.

The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.

[boylan]

This exact thing happened to me when I ran https://www.fatherly.com/ circa 2017. Google just shut down our account without notice. We were spending like $10k/month. It also locked us out of our premium support account, so we couldn't even get anyone there to notice that they'd locked us out.

After about 8 hours, a random Google support tech said it was because we were mining bitcoin, which was laughably untrue. We had CPU usage graphs and logs for the whole time and there was no spike. At around 12 hours, they turned it back on, said it was "misconfiguration of our abuse detection" and gave us like $100 in credit.

Absurd. Say what you will about AWS, they would never do that to a customer without a rep reaching out to you first. I have not trusted GCP since.

[titzer]

Google thinks everything should be replaced with automation.

Remember knowledge cards? Prior to the LLM AI revolution, they had an extraordinarily crappy AI system digest the entire internet to figure out the wrong facts about stuff and then present it to users as solid truth, with no human review and no way to report inaccuracies.

They just don't care. If the task requires a person to look at a thing and tell if it's right, they only do that for like 5 examples and then train a classifier, then deploy said classifier without thinking twice because "at internet scale" or whatever crap.

[dylan604]

Google is the epitome of expecting happy path results to always be the end result. I could absolutely see someone writing this knowledge card system, but then realizing how much work it would be to edit it with some PM not wanting to say the project was a failure and needing serious amounts of human effort to correct and just releasing it as is. Gotta earn those KPIs for that next promotion, and then it's someone else's problem!

[Hikikomori]

Would AWS even care? They didn't when one obs guy was mining shitcoins with unused cpu time. Our security team did care.

[skarz]

$10k a month on a blog??

[anon7000]

Magazine

[Aperocky]

Shouldn't Google answer this if they are unhappy with this incident report? Are we even sure that Railway knows?

[e40]

I seriously doubt Railway knows. That's the MO for Google and others, suspend account without explanation.

[tomComb]

They can't - that would violate the privacy rights of their customer.

They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.

Either way, we need to hear about this from Railway.

[SoftTalker]

The report at this point is pretty much just a timeline of what happened. No explanation of why, no accusations, no blame. A PR piece, to Railway's customers, reassuring them that "we're not ignoring this."

Now the lawyers are huddling. IMO there won't be a lot more said publicly by either side, at least until any threat of lawsuits for damages is settled.

[kyrra]

the Railway PM doesn't say they weren't told. It just sort of glosses over this. I would be interested to know if they were told (or not).

[array_key_first]

I don't think you're typically told why for these things, and it's mostly automated from what I can tell. The automated systems make mistakes but more importantly they're completely opaque. Nobody, not even Google, knows how they work exactly.

[potatoman22]

Google should know why a human accepted the automated suggestion, or if and why there wasn't any human oversight in the first place.

[okanat]

Google knows and wants that there is no oversight. Don't do business with any big tech, if you don't want this kind of incidents.

[stingraycharles]

AWS and Microsoft don’t do this, not like this.

[pocksuppet]

AWS does it, but only for small customers, not for big customers without a human review.

Azure is just so broken they don't have to suspend your account to cause an outage.

[advisedwang]

Google knows why there is no human oversight: because that is expensive (both in terms of the labor doing review and the ongoing fraud likely happening while the human review happens).

[llmslave]

For big accounts, like railway, zero chance this was a handsoff fully automated ban

[x0x0]

That assumes a competent org. If this were aws, I fully believe that. At gcp it's entirely plausible.

[mjcl]

Really? This isn't the first time their automation took down a big customer (UniSuper in 2024) by accident. In that case the automation actually deleted the resources and GCP had to recover them.

[llmslave]

I have worked inside major cloud providers, it doesnt work like this

[croes]

That‘s the point where Google tells you they won’t tell you the exact reason because of security reasons

[apple4ever]

Exactly this, which is the problem with all modern accounts. No person to talk to so you can understand what happened and maybe fix it.

[tcdent]

They most definitely have a person to talk to. They're not the largest Google Cloud user by far, but they are large enough to have human account reps.

[xp84]

And those reps might not be told what the reason is.

[realusername]

They also don't want to tell you because then they have to put rules and cannot ban people arbitrarily.

Giving reasons is putting accountability on Google and they don't want that.

[marginalx]

Who is the "You" in "you haven't addressed the root cause"? If you are asking Railway to spend effort doing this rather than simply moving away from GCP, I'm not sure why they would unless they want to sue GCP to recover damages to brand and long term customer retention.

The moment GCP shut off without any forewarning, its done deal, no need to ask any further questions.

[neya]

Top comments as usual are buried in deep hate for Google, I doubt that will pressure anyone at Railway to address this.