[econner]

It's weird that one of the reasons that you endorse AWS is that you had regular meetings with your account manager but then you regret premium support which is the whole reason you had regular meetings with your account manager.

[dangus]

As a counterpoint, I find our AWS super team to be a mix of 40% helpful, 40% “things we say are going over their head,” 20% attempting to upsell and expand our dependence. It’s nice that we have humans but I don’t think it’s a reason to choose it or not.

GCP’s architecture seems clearly better to me especially if you are looking to be global.

Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare.

GCP’s use of folders makes way more sense.

GCP having global VPCs is also potentially a huge benefit if you want your users to hit servers that are physically close to them. On AWS you have to architect your own solution with global accelerator which becomes even more insane if you need to cross accounts, which you’ll probably have to do eventually because of the aforementioned insanity of AWS account/organization best practices.

[0xbadcafebee]

There's a very large gap between "seems" and reality. GCP is a huge PITA. It's not even stable to use, as the console is constantly unresponsive and buggy, the UX is insane, finding documentation is like being trapped in hell.

Know how you find all the permissions a single user in GCP has? You have to make 9+ API calls, then filter/merge all the results. They finally added a web tool to try and "discover" the permissions for a user... you sit there and watch it spin while it madly calls backend APIs to try to figure it out. Permissions for a single user can be assigned to users, groups, orgs, projects, folders, resources, (and more I forget), and there's inheritance to make it more complex. It can take all day to track down every single place the permissions could be set for a single user in a single hierarchical organization, or where something is blocking some permission. The complexity increases as you have more GCP projects, folders, orgs. But, of course, if you don't do all this, GCP will fight you every step of the way.

Compare that to AWS, where you just click a user, and you see what's assigned to it. They engineered it specifically so it wouldn't be a pain in the ass.

> Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare.

This was an issue in the early days, but it's well solved now with newer integrations/services. Follow their Well Architected Framework (https://docs.aws.amazon.com/wellarchitected/latest/framework...), ask customer support for advice, implement it. I'm not exaggerating when I say this is the best description of the best information systems engineering practice in the world, and it's achievable by startups. It just takes a long time to read. If you want to become an excellent systems engineer/engineering manager/CTO/etc, this is your bible. (Note: you have to read the entire thing, especially the appendixes; you can't skim it like StackOverflow)

[dangus]

I’m sure anyone willing to read through a 1,000 page document can make a well-architected GCP environment, too.

The problem is that no company I’ve ever worked for implemented the well architected framework with their AWS environment, and not one company will ever invest the time to make their environment match that level of quality.

I think what you describe with the web tool to discover user permissions sounds a lot like the AWS VPC Reachability Analyzer which I had to live in for quite a while because figuring out where my traffic was getting blocked between an endless array of AWS accounts and cross-region transit gateways was such a nightmare that wouldn’t exist with GCP global VPCs and project/folder based permissions.

I don’t like the GCP console, but I also wouldn’t consider a lot of the AWS console to be top tier software. Slow/buggy/inconsistent are words I would use with the AWS console. I can concede that AWS has better documentation, but I don’t think it’s a standout, either.

[secondcoming]

GCP's UI sure is dog slow. I once filled in that 'How would you rate...' prompt that sometimes appears about Instance Group management via the UI and it seems they later addressed the issue.

[danpalmer]

Similar to my experience with the two. We didn't have regular meetings with our GCP account manager, but they did help us and we had a technical support rep there we were in contact with sometimes. We rarely heard from anyone at AWS, and a friend had some horror stories of reporting security issues to AWS.

Architecturally I'd go with GCP in a heartbeat. Bigquery was also one of the biggest wins in my previous role. Completely changed out business for almost everyone, vs Redshift which cost us a lot of money to learn that it sucked.

You could say I'm biased as I work at Google (but not on any of this), but for me it was definitely the other way around, I joined Google in part because of the experience of using GCP and migrating AWS workloads to in.

[SkiFire13]

> Every organization I’ve ever witnessed eventually ends up with some kind of struggle with AWS’ insane organizations and accounts nightmare.

What are these struggles? The product I work on uses AWS and we have ~5 accounts (I hear they used to be more TBF) but nowadays all the infrastructure is on one of them and the other are for some niche stuff (tech support?). I could see how going overboard with many accounts could be an issue, but I don't really see issues having everything on one account.

[sylens]

I like AWS, but Organizations was something that was retrofit onto the account model versus being part of the original design. GCP had second mover advantage in this area.

The way to automate provisioning of new AWS accounts requires you to engage with Control Tower in some way, like the author did with Account Factory for Terraform.

[blitzar]

AWS makes the account model feel retrofit versus being part of the original design and 5 years later someone retrofit the organisations onto that before they they added 90% of the products into any square round hole they could find.

[sleepychu]

We were saved by the bell when they announced the increased account limit for S3 buckets (1M buckets, now, 1k I think before).

Just before they announced that I was working on creating org accounts specifically to contain S3 buckets and then permitting the primary app to use those accounts just for their bucket allocation.

AWS themselves recommend an account per developer, IIRC.

It's as you say, some policy or limitation might require lots of accounts and lots of accounts can be pretty challenging to manage.

[tech2]

I have to ask, because wow that's a lot of buckets, but what kind of activity requires breaching even a 1,000 bucket limit per account?

[sleepychu]

Simplistic tenant isolation and cost tracking :-)

I know there are other solutions to this particular problem but this model is extremely easy to reason about. When the application accesses tenant objects or delegates that access with pre-signed URLs it is doing so with ephemeral credentials that literally could not access the objects in another tenancy.

That and a similar DB isolation, allows most of our handlers to be very simple as far as tenant isolation goes.

[dangus]

5 accounts would be heaven if that could be my environment.

I have almost 40 AWS accounts on my login portal.

Two accounts per product, one for development environments and one for production environments, every new company acquisition has their own accounts, then we have accounts that solely exist to help traverse accounts or host other ops stuff.

Maybe you don’t see issues with everything in one account but my company would.

I don’t really think they’re following current best practices but that’s a political issue that I have no control over, and I think if you went back enough years you’d find that we followed AWS’ advice at the time.

[UltraSane]

Global VPCs are very nice but they feel like a single blast radius.

[dangus]

Whether or not your VPC can have subnets in multiple regions is entirely unrelated to security.

[UltraSane]

I meant failure blast radius. Having isolated regions is a core part of the AWS reliability design. AWS has had entire regions fail but these failure have always been isolated to a single region. Global VPCs must rely on globally connected routers that can all fail in ways AWS regional VPCs can't.

[ses1984]

If you need global HA to the extent that you're worried about global VPC failure modes, you're going to have to spend a lot of effort to squeeze uptime to the max regardless of where you deploy.

Undersea cable failures are probably more likely than a google core networking failure.

In AWS a lot of "global" things are actually just hosted in us-east-1.

[easton]

On the other hand, when they say something is in us-west-2 they mean it, so if another region has an outage your workloads aren't impacted unless your code is reaching out to that region.

Guessing that's similar on the other clouds.

[dangus]

I believe you’re likely misunderstanding Google’s architecture.

The routing isn’t centralized, it’s distributed. The VPCs are a logical abstraction, not a centralized dependency.

If you have a region/AZ going down in your global VPC, the other ones are still available.

I think it’s also not that much of an advantage for AWS to be able to say its outages are confined to a region. That doesn’t help you very much if their architecture makes architecting global services more difficult in the first place. You’re just playing region roulette hoping that your region isn’t affected. Outages frequently impact all/multiple AZs.

[UltraSane]

Regions and Availability Zones are very different things. Regions are much more isolated.

[sandorscribbles]

googlers in their infinite wisdom have built a startup ecosystem for gcp that assigns "startups" to entry level new hires who are scrambling to figure out how to be managers of accounts while learning how to talk to humans, because googlers are generally not used to interacting with humans, just code, and that is the result of the programmatic hiring/screening process. each newly hired 20-something is also assigned 3,000 gcp accounts to manage.

what is the engineering used to determine a weak startup from a growing company you ask? well....googlers again use random numbers not logic (human interaction avoidance firewall) to determine that and set the floor at $30M "publicly declared investment capital". so what happens when you the gcp architect consultant hired to help this successful startup productionalize their gcp infra but their last round was private? google tells the soon to be $100M success company they are not real yet.....so they go get their virtual cpu,ram,disk from aws who knows how to treat customers right by being able to talk to humans by hiring account managers who pick up the phone and invite you to lunch to talk about your successful startup growing on aws. googlers are the biggest risk factor to the far superior gcp infrastructure for any business, startup or fortune 10.

[sharts]

having used gcp, i don’t think anything they do is far superior

[unsnap_biceps]

If you spend enough (or they think you'll spend enough), you'll get an account manager without the premium support contract, especially early in the onboarding

[necubi]

Or if you’re a newish startup who they hope will eventually spend enough to justify it.

[mannyv]

If you know what you're doing you don't need AWS support.

We add support when we want to do something new, like MediaTailor + SSAI. At that point we're exploring and trying to get our heads around how things work. Once it works there's no real point in support.

That said, you need to ask your account manager about (1) discounts in exchange for spend commitments, and (2) technical assistance. In general we have a talk with our AM when we're doing something new, and they rope in SMEs from the various products for us.

We're not that big, and I haven't worked for large companies, and it's always been a mystery to me why people have problems dealing with AWS. I've always found them to be super responsive and easy to get ahold of. OTOH we actually know what we're doing technically.

Google Cloud, OTOH, is super fucked up. I mean seriously, I doubt anyone there has any idea WTF is happening or how anything works anymore. There's no real cohesion, or at least there wasn't the last time I was abused by GCP.

[nucleardog]

> That said, you need to ask your account manager about (1) discounts in exchange for spend commitments, and (2) technical assistance.

Depending what precisely you mean by the second one, you may not even need an AM/support for that.

They won't help me use the platform, but they will still address issues with the platform. If you run into bugs, things not behaving how they're documented, or something that simply isn't exposed/available to customers they seem to be pretty good about getting it resolved regardless of your spend or support level.

(On my personal account with minimal spend, no AM, and no support... I've had engineers from the relevant teams email me directly after submitting a ticket for issues.)

So yeah, "if you know what you're doing" you probably don't even need the paid-for support.

[darth_avocado]

> If you know what you're doing you don't need AWS support.

Hard disagree. I have to engage with AWS support almost once every 6 months. A lot of them end up being bugs identified in their services. Premium support is extremely valuable when your production services are down and you need to get them back up asap.

[h1fra]

I never got this in the comparison of aws between gcp. Why do people need direct support that much? In 8 years, I had to reach out to GCP maybe twice and still got an answer anyway.

[mnahkies]

We've only raised a handful of support cases with GCP the past 5 years, but we happened to raise one this week and they've put us onto a preview feature that solves the problem we were facing - I'm suddenly wondering if we should be trying our luck with support more often instead of figuring it out ourselves.

[gizzlon]

Preview, as in no sla [1] and they could cancel it or remove features any time in the future?

1: yes, I know the sla’s are usually a joke

[travisd]

I found two separate bugs in GCP products. One with gVisor where it would sometimes null-truncate large network packets (this was very hard to diagnose – why is my JSON full of null bytes?) and one where Cloud Run broke sudo sporadically (sudo in a FaaS is definitely niche, I had essentially containerized a very old application written by undergraduates).

Both times they were serious production bugs that took at least a week to resolve, though I only had the lowest tier of support package.

[JoeBOFH]

With my experience it’s the edge cases. The few times I had to reach out to AWS support were due to some weird edge case we couldn’t fix but AWS had to. And having a rep involved made it so much smoother.

[h1fra]

I had to reach AWS because of a bug in Aurora last year; they replied quickly but said that they couldn't understand the bug...

[sandorscribbles]

if you need to bump a quota above the predetermined range of what googlers think is "normal" usage (which is far too low to run anything at scale)you have to talk to a human to negotiate the quota bump. why? because googlers in their infinite engineering wisdom use "gcp quotas" not as a cost optimization guardrail for customers benefit, but to inform google on when and how much metal they need to buy for their datacenter region you are running in.

[verst]

I have to defend the Googlers here (I work at a different hyperscaler). Teams / services need to optimize their COGS. That means optimizing infrastructure cost. A lot of pay as you go service may not have any base cost to customers but they require some infrastructure to be provisioned. Without quotas you can have a lot of provisioned infrastructure which does not produce any revenue to even collectively break even. Just yesterday this a decision we evaluated again in my team. As a team we cannot afford an unlimited quota - both because of what that would do to our bottom line and because we can't necessarily obtain all the quotas we need ourselves to provision enough capacity for our dependencies. It's a difficult trade off requiring manual intervention.

[sandorscribbles]

i may have not emphasized enough how important quotas are for customers. quotas are very important guardrails for orgs that ensure that newly hired engineer who wants to "test drive the cloud" by running a BigQuery tutorial they found on github, gets stopped before they burn $10k in an afternoon. however, quotas on gcp are there for the benefits of google and not geared towards the customers. first there is an ever expanding tree of potential quotas complicating production rollouts of infra and second they are all set insanely low so even the smallest POC gets blocked. requesting a small increase routes the quota through software and auto-approval, requesting a quota that allows for a production workload? 3 weeks + help from your account rep, if google has blessed you the privilege of being allowed to talk with a human googler. no account rep you say, well your production workload can just wait around for google support to potentially acknowledge your existence.

[dielll]

One day, you will need support and when you do, you will realise why every week there's a top voted post on HN on someone complaining about not reaching Google Support

[rco8786]

The regret was about the cost of the premium support