[jiggawatts]

I’m very curious to hear more details!

Did you use reservations to reduce costs?

Was it a lift and shift with VM configs staying as-is? (I’ve seen a lot of empty 1TB “app” drives burning money in the cloud!)

You complain about PaaS services, but I can’t imagine 8 data centres worth of stuff being converted to PaaS in hurry!

[gryf]

Cost savings are mostly consolidation, scaling down stuff we don't need (we have peak hours) and migrating stuff to kubernetes and packing it tight.

[jiggawatts]

Despite that you were paying 8x the previous amount!?

How is that possible?

E.g.: With Kubernetes and AWS you ought to be able to use clusters with a base of "reserved" capacity plus spot pricing for peak hours on top, right?

From what I've seen (in my limited experience), that should reduce costs for most orgs, not increase them!

[gryf]

I have identified the limits of human incompetence.

Please someone hire me so I don't have to live through this nightmare any longer.

[yjftsjthsd-h]

> How is that possible?

TLDR: AWS is really expensive compared the equivalent compute elsewhere, to the point of overwhelming its advantages.

---

Long answer that I wrote before realizing how long it was:

I'm not the person you're replying to, I don't know the exact details, I don't know their company. However:) I can tell you that AWS is a factor more expensive than renting or owning your own metal. They (claim to) offset this by 1. taking care of management for you so you need fewer ops people, and 2. letting you scale up and down as needed. The first is plausibly legitimate, but really depends on the size of your compute and the size of your human teams (and how much benefit you get from AWS-managed offerings). The second can help, but only if you've got really spiky workloads, that are only really running a tiny fraction of the time, with absolutely tiny base load. Scaling up and down helps reduce your AWS bill compared to not scaling up and down on AWS, of course, but it doesn't help that much when their elastic offerings are that much more expensive. Say, for the sake of argument that you can run most of your EC2 instances just 6 hours a day, during peak hours. That lets you win... if, and only if, EC2 instances are less than 4x the cost of just owning your own bare metal machines. I've gone so far as trying to price out using spot instances for CI work - just selectively, when they were the cheapest! - to augment instances on Hetzner Cloud. Guess what? They're so expensive that even at spot prices EC2 is a factor more expensive.

[jiggawatts]

Really?

I just priced up a 128-core (dual EYPC) Dell server with specs comparable to a matching Azure cloud server (HB120rs_v3), and the "1 year reserved" price for Azure came out to about the same as the purchase price over 3 years. The 3-year reserved price is about the same as the purchase price over 4 years. There's a new 5-year reservation option, which is the equivalent of owning the hardware for 7 years. Spot pricing is the equivalent of amortizing the purchase price over 12 years!!

Meanwhile, using the cloud, you can upgrade to the 9004 series in just a few months, not 12 years from now. And then whatever comes after the 9004 series in like 2 years, not 5, 7, or 12.

So it looks like that at the larger scales, the pricing is very directly competitive with on-premises hardware.

Consider that the cloud hosts include most basic operations costs, such as cooling, electricity, data centre floor space rental, the SFP ports and cabling, etc, etc...

Speaking of which, a quick back-of-the-envelope calculation is that a 128 core server will cost about $20K in cooling and electricity over its lifetime.

Note that the Azure HB120rs_v3 sizes come with 200 Gbps InfiniBand "just thrown in" for laughs. Try pricing that up some time, in case you want to build your own hyperconverged infrastructure!

Admittedly, at the smaller sizes, Azure and AWS are less competitive, but you do get flexibility, automation, and a bunch of other stuff that's difficult and expensive at scale on-prem.

[tester756]

it feels like the "refactor" was the way to improve

[gryf]

I love reading people saying this. Try a couple of thousand DB tables evolved organically. Nope not happening.

[tester756]

I meant that what you did was "refactor" - rethinking the whole mess

So it wasn't purely cloud/renting racks issue