[jjoonathan]

Sure, but it it has to be memeable. Imagine technical manages bickering in front of their managers.

"Let me get this straight, Google canceled a service and you didn't see it coming?"

^ this needs a comeback.

"Let me get this straight, you ran into another AWS scaling limit and you didn't see it coming?"

or something.

[abraxas]

If I had a dime for every time I see a convoluted architecture in our code that only exists due to the arbitrary hard limits in AWS I'd be able to quit and never again worry about arbitrary AWS limits.

[jjoonathan]

I'm pretty sure they aren't arbitrary at all and are mostly due to managed and serverless offerings backed by duct tape and chewing gum. On a few occasions I have been left with all-but-proof that behind the "let us handle scaling for you" smoke and mirrors is a bash script on an EC2 instance sized by guesswork and hardcoded until you complain, at which point support might be able to shuffle it between instance types to eek out a bit more vertical scaling.

[abraxas]

Yes, it's part this and part "let's get rid of the noisy neighbour issue by making everyone whisper" design philosphy. This just arrogantly pushes the scaling challenges back down to the customers who are then forced to buy more AWS services to build workarounds for the scaling limitations in front of them. For Amazon it's a win-win though and they'll keep doing this as long as c-suite fools keep buying.

[jen20]

I spent the first months 2020 building out a database-as-a-service offering that runs in AWS, Azure and GCP (think Cockroach Cloud or MongoDb Atlas model, but for a different database).

That was an instructive project - building the same service in three clouds tells you a lot both about:

- The quality and completeness of foundational services (identity, networking, compute, storage)

- The tooling ecosystem (the quality of the Packer builders and Terraform providers [1] in our case)

- How helpful (or existent) support is, which ranged from an account manager telling us up-front “here’s the way to avoid hitting limits for your design” to not being able to talk to a human at all throughout the entire project, and thus having to phase in beta customer onboarding for that cloud because of the arbitrary limits.

At some point that team should write a full retrospective on this.

[1]: Disclaimer - I have worked on both Packer and Terraform in the past at HashiCorp.

[abraxas]

So, what are your experiences?

[jen20]

These are personal opinions, based on the project I outlined (and not what I work on now, necessarily!).

Technically:

- Google has the most reliable network, compute and storage (for a given size).

- AWS has the only comprehensible security model for identity, although it's still not complete (e.g. I can't grant a role assigned to an instance profile permission to `DescribeInstances` for itself only). I strongly believe IAM is the crown jewel of AWS, - but wish it would be completed to it's own potential.

- Google has the best "organisations" structure overall, though AWS Organisations is vastly improved over what it used to be.

- Azure's model for network peering between networks in different tenants is complete crazy town and will certainly result in outages when a customer disables the service account required to maintain it.

- Provisioning times in Azure are wildly variable - provisioning a VM with the same image in the same zone often had minutes of difference between fastest and slowest. The other two are much more consistent.

- The Terraform provider for Google is missing many data sources, and almost every type of resource we used needed patching in some important way.

- We had to build "surrogate" Packer builders for Google and Azure to make automation of scratch-build ZFS-on-root Ubuntu images with our platform customisations. I built the AWS version of that builder originally, so that was not much of a surprise.

From a support perspective:

- AWS and Azure were very willing to work with us in getting our service up and running even though we weren't spending a huge amount in the development phase, and it was easy to get in touch with someone to explain what we were doing and request advice.

- It was impossible to speak to a human at Google. Experiencing the kafkaesque automated account policies (e.g. "you can have enough cores to actually bring up a database cluster when you pay your invoice but we haven't issued an invoice yet because your account isn't a month old, and no despite being a company with a multi-year trading history you can't just put money on deposit to prove trustworthiness") actually prompted me to move my personal accounts off GSuite in case a problem ever arose.

Sadly the technical excellence of GCP in several important areas did not (for me) make up for the fact that they are impossible to work as a small business doing something that is not strictly happy path.