[cbushko]

This resonates with me so much as I was in this position 3 years ago.

My boss came to me and said "we want to move from AWS to GCP, using kubernetes and not have it be a mess like we have now"

I was new, senior, free from ownership. I chose to stay as a team of 1 so that I wasn't burdened with fires and meetings. It freed me up to put my head down and get the work done.

The grind was learning GCP, kubernetes, terraform, the platform/product, gathering requirements from several teams and mentoring people that would pop into the project as their time allowed.

I won't lie as learning new things was great. The project was partially greenfield as it was a platform migration and not a product rewrite. There were a lot of constraints & debt to deal with.

Maybe the new employee churn is one of the reasons that most startups get so much done... the new people are not burdened by ownership and being the subject matter experts. They are free to do things, good or bad. :)

[otabdeveloper4]

> we want to move from AWS to GCP

Out of the frying pan and into the fire.

[kakkan]

Why do you think GCP is worse than AWS? I've used GCP and found it quite easy to start with and the managed instances was a breeze to set up and deploy stuff. I've done it for only small businesses and they were happy with the results too and the easy to use console made things easier to navigate. Maybe the trouble with GCP comes with scale?

[otabdeveloper4]

I'm strictly against all "cloud" lock-in, but if you're going down that path despite all caveats, then using the #1 cloud provider makes more sense than a third-place competitor from a company that's best known for not having customer service.

[sangnoir]

Kubernetes is as anti-lock-in as you can get when it comes to cloud platforms. Which makes sense when you think it's the #3 competitor who mostly sponsors it - on the whole, lock-in benefits the market leader, so commoditization works to upstarts advantage.

[kelp]

I recently worked at a single digit billion dollar valuation SaaS provider and we had to run everything on AWS, GCP and Azure, to be where the customers were.

Total cloud spend was in 2 digit millions per year.

Many hundreds of k8s clusters, all using each providers k8s managed service.

My general high level take is this:

* AWS

Products mostly work quite well. Their UX is terrible and very inconsistent between products. But things mostly work, and their support is generally excellent, especially when you have a TAM (Technical Account Manager) who you can reach out to at any time. They will help you navigate any support issues, and ensure you get a quick resolution. But AWS stuff is generally very complex, overly complex in my opinion.

Also at scale API rate limits just kill you, and AWS continues to struggle to provide any real visibility into API rate limits. I'm told that large services internal to AWS have to work around this by building their own caching layer in front of any control plane APIs. Also the api limits are at the account boundary, so you have to start doing account sharding.

GCP:

Has better UX, more consistent experience. Much better visibility into limits and things like that. You still probably end up having to do project sharding at scale to work around limits. We continued to run into a lot of issues with GCPs load balancers blackholing traffic, and it was never really resolved. I think their support is generally worse than AWS. I've had substantially worse support experiences with GCP than AWS at 2 companies now. Mostly support not being particularly useful, and not resolving the issue. As a customer it feels like Google has struggled to adapt their culture to the needs of real enterprise customers. AWS has this much more figured out. But my GCP account teams have always been quite good.

Azure: The worst of all 3. Lots and lots of issues with their k8s service, low limits on the number of nodes you could deploy. k8s 1.18 supposedly fixed a lot. But their answer was always just that you have to upgrade. Their support engineers were frequently gave us actively bad advice that would have resulted in full customer outages if we'd followed it. The engineers on my team fortunately didn't listen and instead figured out how to fix issues themselves. Also frequently ran into various limits and you need to start thinking about subscription sharding.

Of the 3, I personally prefer using and interacting with GCP. But their support or is still certainly worse than AWS. That said, I always felt bad for our TAMs at AWS. It felt like a non-stop job where you had too be available at all hours to deal with unhappy customers. They usually gave us fantastic support, but I wonder at what personal cost.

[kakkan]

Thank you, appreciate the detailed answer.

[jen20]

> Why do you think GCP is worse than AWS?

Because the identity model is worse than IAM, and the quota system is insane. The global VPC networking being nicer doesn't make up for this, unfortunately.

[cbushko]

The fire is 1/3 of the cost of the frying pan.