[oersted]

As a CTO of a number of small startups, I am still struggling to understand what exactly AWS and other cloud providers give you to justify the markup.

And yes we’ve been heavy users of both AWS and Google Cloud for years, mainly because of the credits they initially provided, but also used VMs, dedicated servers and other services from Hetzner and OVH extensively.

In my experience, in terms of availability and security there’s not much difference in practice. There are tons of good tools nowadays to treat a physical server or a cluster of them as a cloud or a PaaS, it’s not really more work or responsibility, often it is actually simpler depending on the setup you choose. Most workloads do not require flexible compute capability and it’s also easy and fast to get it from these cheaper providers when you need to.

I feel like the industry has collectively accepted that Cloud prices are a cost of doing business and unquestionable, “nobody ever got fired for choosing IBM”. Thinking about costs from first principles is an important part of being an engineer.

[aurareturn]

When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it.

Or you need to restore your Postgres database and you find out that the backups didn't work.

And finally you have a brilliant idea of hiring a second $150k/year dev ops admin so that at least one is always working and they can check each other's work. Suddenly, you're spending $300k on two dev ops admins alone and the cost savings of using cheaper dedicated servers are completely gone.

[isoprophlex]

When your AWS bill suddenly spikes to $69k because some data science intern left a huge gpu backed EC2 instance running in ap-southeast-2 with a misconfigured auto-scaling group, and your CTO is at a "digital transformation" conference, and you have hundreds of angry investors asking why your burn rate tripled, you’ll get it.

Or you need to debug why your Lambda function is throttling and you find out that the CloudWatch logs were never properly configured and you’ve been flying blind for three months.

And finally you have a brilliant idea of hiring a second $150k/year AWS solutions architect so that at least one person can actually understand the bill and they can check each other’s Terraform configs. Suddenly, you’re spending $300k on two cloud wizards alone and the cost savings of "not managing your own infrastructure" are completely gone.

The snide rebuttal basically writes itself.

[bdangubic]

each rebuttal is about hiring shitty/icompetent people which also is same if you gonna manage your own (except of course much, much worse) - so not much or a rebuttal

[array_key_first]

AWS facilitates the shittiness and incompetence because it, supposedly, handles all this stuff for you so you don't have to think about it.

Except - wait, you do have to think about, because of course you. So the promise of AWS is gone.

[tiew9Vii]

> When your cheap dedicated server goes down and your admin is on holiday and you have hundreds of angry customers calling you, you'll get it.

Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted.

Or your cloud provider wipes your account and you are a $135B pension fund [1]

Or your cloud portfolio is so big you need a "platform" team of multiple devops/developer staff to build wrappers around/package up your cloud provider for you and your platform team is now the bottleneck.

Cloud is useful but it's not as pain free as everyone says when comparing with managing your own, it still costs money and work. Having worked on several cloud transformations they've all cost more and taken more effort than expected. A large proportion have also been canned/postponed/re-evaluated due to cost/size/time/complexity.

Unless you are a big spender with dedicated technical account manager, your support is likely to be as bad as a no name budget VPS provider.

Both cloud and traditional hosting have their merits and place.

[1] https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...

[aurareturn]

  Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted.

https://news.ycombinator.com/item?id=42365295

https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_ca...

[grey-area]

It is statistically far more likely that your cloud service will go down for hours or days, and you will have no recourse and will just have to wait till AWS manage to resolve it.

[n4r9]

I suspect that this is really about liability. When AWS goes down you can just throw up your hands, everyone's in the same boat. If your own server goes down you worry that your customers doubt your competence.

It's actually kinda frustrating - as an industry we're accepting worse outcomes due to misperceptions. That's how the free market goes sometimes.

[PaulKeeble]

Nobody gets fired for hiring IBM. This is the new version, when you go down because AWS did its someone else’s fault. Of course AWS will compare their downtime to industry standards for on premise and conclude they are down less often. On Premise engineers can say until they are blue that their downtime is on a Sunday at 3 am because it doesn't impact their customers it doesn't seem to matter.

[BrandoElFollito]

On the other hand when Google mail gies down, I am happy to be in yhe same boat as 2 B people, waiting for the page to refresh.

As opposed to be with the small provider round the corner who is currently having a beer and will look at that tomorrow morning.

Now - I am in the phase where I ap seriously considering to move my email from Google to a small player in Europe (still not sure who) so this is what may ultimately be my fate :)

[theshrike79]

When us-east-1 goes down, half the internet goes down with it.

Customers call and complain about downtime, I can just vaguely point at everything being on fire from Facebook to Instagram to online banking sites.

They get it.

When the self-hosted server fries itself, I'm on the hook for fixing it ASAP.

[nurettin]

I guess you sip coffee, watch true crime on yt and tell everyone there is a global outage while aws us-east-1 fixes it compared to burning the midnight oil when you are the one fixing it. Totally worth paying 10x when that happens.

[aurareturn]

The difference is that if AWS goes down, I know for a fact that it'll be back up without me doing anything.

If my own dedicated server goes down, I'm going to need to call my admin at 3am 10 times just to wake him up.

[jrk]

You know that AWS will come back up. You definitely don’t know whether your own instances will come back or if you’ll need to redeploy it all.

[BrandoElFollito]

Why do you assume that the small dedicated server has a higher probability to come back?

[KetoManx64]

If youe admin isn't competant enough to setup logging or notifications, how is it going to be better when your Cloud VM runs out of storage or doesn't reboot properly due to AWS swapping out hardware?

[_rwo]

> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin (...)

in my experience you always need a "Devops team" to operate all that cloud stuff; so to paraphrase - suddenly you're spending $400k on three devops to operate $500k cloud

I think The Promise behind the cloud was you just pay for the service and not worry about it, but in practice you need some team to maintain it

[oersted]

After 8 years operating like this, I have had approximately the same number of critical outages in standard Cloud as with these providers.

One included a whole OVH building burning down with our server in it, and recovery was faster than the recent AWS and Cloudflare outages. We felt less impotent and we could do more to mitigate the situation.

If you want to, these providers also offer VMs, object storage and other virtualized services for way cheaper with similar guarantees, they are not stuck in the last century.

And I don’t know how people are using cloud, but most config issues happen above the VM/Docker/Kubernetes level, which is the same wether you are on cloud or not. Even fully managed database deployments or serverless backends are not really that much simpler or less error-prone than deploying the containers yourself. Actually the complexity of Cloud is often a worse minefield of footguns, with their myriad artificial quirks and limitations. Often dealing with the true complexities of the underlying open-source technologies they are reselling ends up being easier and more predictable.

This fearmongering is really weakening us as an industry. Just try it, it is not as complex or dangerous as they claim.

[jrk]

It is not only not that much more complex, it is often less complex.

Higher-level services like PaaS (Heroku and above) genuinely do abstract a number of details. But EC2 is just renting pseudo-bare computers—they save no complexity, and they add more by being diskless and requiring networked storage (EBS). The main thing they give you is the ability to spin up arbitrarily many more identical instances at a moment’s notice (usually, at least theoretically, though the amount of the time that you actually hit unavailability or shadow quotas is surprisingly high).

[aurareturn]

I'm a geek and I like to tinker with hardware. I want to maximum my $/hardware and have built a ton of DIY computers myself since I was young. I'm all about getting the most hardware for the money.

But I'd like to sleep at night and the cost of AWS is not a significant issue to the business.

[oersted]

That’s fair enough but that’s a luxury position, if costs are not concern to you then there’s not much point in discussing the merits of different methods to manage infrastructure efficiently.

And yes of course such costs are nothing if you are thinking of $300K just on a couple sysadmins. But this is just a bizarre bubble in a handful of small areas in the US and I am not sure how it can stay like that for much longer in this era of remote work.

We built a whole business with $100K in seed and a few government grants. I have worked with quite a few world-class senior engineers happily making 40K-70K.

[aurareturn]

Don't get me wrong. If I'm starting a brand new business with my own money and no funding, I'd absolutely buying a cheap dedicated instance. In the past, AWS gave out generous credits to startups/new businesses. This is no longer the case.

Once my business requires reliability and I need to hire a dedicated person to manage, I'd absolutely move to the cloud. I personally like Digital Ocean/Render.

[bjourne]

What prevents an EC2 instance from going down in exactly the same way? Other hosting providers offer automatic backup too - it's not an AWS exclusive feature.

[aurareturn]

Nothing. It's just that I'm not the one responsible to fix it at 3am.

[gizzlon]

So if you app enters a crash-loop and fails to start an AWS engineer comes in and fixes it? Because that has not been my experience..

The truth is that there's still a lot of things you have to handle, including cloud bugs and problems. And other problems you don't have to think about anymore, especially with fully managed, high-level PaaS- like services.

I ran a cloud backend service for a startup with users, using manged services, and we still had an on-call team. The cloud is not magic.

[bjourne]

Then who is responsible to fix it?

[aurareturn]

AWS if it's AWS' fault.

[bjourne]

You do realize that a server can "go down" for many other reasons than "the intern pulled the plug on it", right?

[Perz1val]

That's even worse when AWS goes down and myth of it never going down should be more than shattered by now

[jjav]

> And finally you have a brilliant idea of hiring a second $150k/year dev ops admin

The initial impression that we don't need to hire many people because AWS takes care of everthing, fades away pretty quick.

You still need to hire the same people, they just do the same things in a different way.

[utopiah]

> your admin is on holiday and you have hundreds of angry customers

Single human point of failure for something tied to the bottom line is NOT a technical problem, it's management.

[esseph]

Sure, but AWS has more downtime than I do :-)

[fragmede]

If we assume that you're a human being that sleeps, say 8 hrs/day, and not an LLM, that leaves you with 16 hours of consciousness a day, for an uptime of 66%. That's upsidedown nines. You don't even crack one nine of uptime. If we assume you've a diet of meth and cocaine, and only sleep 2 hours a day, that still puts you at only like 92% uptime.

[esseph]

> If we assume that you're a human being

I'll have you know I am a cantaloupe, you insensitive clod!

[aurareturn]

Yes but you go on holidays but AWS does not.

[veeti]

You guys aren't relying on only RDS backups, right? Something off AWS, right?

[juliusceasar]

When your system goes down on AWS and your AWS admin is on holiday, you'll have the same problem.

What is your point?

[wink]

Every team I have worked on so far, if using AWS you had 50-100% of the developers with the knowledge and credentials (and usually the confidence) to troubleshoot/just fix it/replace it.

Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware.

I'm not even one of the "cloud is so great" people - but it you're generally doing software it's actually a lot less friction.

And while the ratio of cost difference may sound bad, it's generally not. Unless we're talkign huge scale, you can buy a lot of AWS crap for the yearly salary of a single person.

[tpxl]

You said developers have the knowledge and credentials (and thus the work) of managing your infra, and a moment later basically asserted you're saving money on the salary for the sysadmin. This is the actual lie you got sold on.

AWS isn't going to help you setup your security, you have to do it yourself. Previously a sysadmin would do this, now it's the devs. They aren't going to monitor your database performance. Previously a sysadmin would do this, now it's the devs. They aren't going to setup your networking. Previously a sysadmin would do this, ...

Managing hardware and updating hosts is maybe 10% of the work of a sysadmin. You can't buy much on 1/10th of a sysadmins salary, and even the things you can, the quality and response time are generally going to be shit compared to someone who cares about your company (been there).

[wink]

Yes, please continue explaining the job I did in the past to me.

It doesn't change anything, especially as I did not blatantly argue cloud=good,hardware=bad. That is a completely different question.

My point is that given some circumstances, you need a lot less specialized deep knowledge if all your software just works[tm] on a certain level of the stack upwards. Everyone knows the top 1/3 of the stack and you pay for the bottom 2/3 part.

I didn't mean to say "let's replace a sysadmin with some AWS stuff", my point was "100k per year on AWS makes a lot of small companies run".

Also my experience was with having hardware in several DCs around the world, and we did not have people there (small company, but present in at least 4 countries) - so we had to pay for remote hands and the experience was mostly bad . Maybe my bosses chose bad DCs, or maybe I'd trust sysadmins at "product companies" more than those working as remote hands at a hoster...

[graemep]

> Every team I have worked on so far, if using AWS you had 50-100% of the developers with the knowledge and credentials (and usually the confidence) to troubleshoot/just fix it/replace it.

is that because they were using AWS so hired people who knew AWS?

I would personally have far more confidence in my ability to troubleshoot or redeploy a dedicated server than the AWS services to replace it.

> Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware.

There are lots of options for renting dedicated hardware, that the service provider will maintain,. Its still far cheaper than AWS. Even if you have redundancy for everything its still a lot cheaper.

[aurareturn]

I don't have an AWS admin. I assume a $2.4 trillion dollar company always has dev ops on call?

[mgaunard]

What AWS gives you is the ability to spin up dozens if not thousands of hosts in a single click.

If you run your own hardware, getting stuff shipped to a datacenter and installed is 2 to 4 weeks (and potentially much longer based on how efficient your pipeline is)

[oersted]

What really needs thousands of hosts nowadays? Even if you have millions of users. Computers are plenty fast now and leveraging that is not any harder if you choose the right stack.

And even if you are building with microservices, most standard servers can handle dozens in a single machine at once. They are all mostly doing network calls with minimal compute. Even better actually if they are in the same host and the network doesn’t get involved.

If you want to, there are simple tools to hook a handful of them as a cluster and/or instantly spawn extra slightly costlier VMs in case of failure or a spike in usage, if a short outage is really a world-ending event, which it isn’t for almost every software system or business. These capabilities have not been exclusive to the major cloud providers for years.

Of course we are generalizing a lot by this point, I’d be happy to discuss specific cases.

[tempestn]

If you own your own hardware, but you can provision a leased dedicated server from many different providers in an hour or three, and still pay far less than for comparable hardware from AWS.

[mgaunard]

That sounds like a good deal, what providers offer this?

[graemep]

OVH, Hetzner, Scaleway. Lots of smaller providers. Most people who offer dedicated servers do this.

[noir_lord]

Which is an awesome capability, if you need it.

I suspect that if you broke projects on AWS down by the numbers, the vast majority don't needed it.

There are other benefits to using AWS (and drawbacks) bit "easy scaling" isn't just premature optimisation because if you build something to do something it's never going to do that's not optimisation it's simply waste.

[mgaunard]

They need it at the beginning to get started quickly, then they don't but don't bother moving out.

Not too different from how many other lines of business get their clients in the door.

[continuational]

But on Hetzner, you can usually get a dedicated server installed and ready tomorrow.

[giuseony]

Even sooner than that. I bought a dedicated server on auction and it was ready 2h after the payment.

[h33t-l4x0r]

Hetzner is oversold. It's not appropriate for production in the same sense that EC2 obviously is. It's fine for staging though.

[nake89]

Out of curiosity, how is a dedicated server oversold?

[tempestn]

I'm not familiar with Hetzner personally, but maybe they mean the uplink? I've found that with some smaller providers, advertising 10Gbit, but you rarely get close to that speed in reality.

[paranoidrobot]

> “nobody ever got fired for choosing IBM”

Agreed, there's definitely a heavy element of that to it.

But, at the risk of again being labelled as an AWS Shill - there's also other benefits.

If your organisation needs to deploy some kind of security/compliance tools to help with getting (say) SOC2 certification - then there's a bunch of tools out there to help with that. All you have to do then is plug them into your AWS organisation. They can run a whole bunch of automated policy checks to say you're complying with whatever audit requirements.

If you're self-hosting, or using Hetzner - well, you're going to spend a whole lot more time providing evidence to auditors.

Same goes with integrating with vendors.

Maybe you want someone to load/save data for you - no problems, create an AWS S3 bucket and hand them an AWS IAM Role and they can do that. No handing over of creds.

There's a bunch of semi-managed services where a vendor will spin up EC2 instances running their special software, but since it's running in your account - you get more control/visiblity into it. Again, hand over an AWS IAM Role and off you go.

It's the Slack of IAAS - it might not be the fastest, it's definitely not the cheapest, and you can roll your own for sure. But then you miss out on all these integrations that make life easier.

[kamaal]

>>As a CTO of a number of small startups, I am still struggling to understand what exactly AWS and other cloud providers give you to justify the markup.

If you are having a company that warrants building a data center, then AWS does not add much.

Other wise you face the 'if you want to build apple pie from scratch, you need to first invent the universe' problem. Simply put you can get started right on day one, in a pay as you go model. Like you can write code, deploy and ship from the very first day, instead of having to go deep down the infrastructure rabbit hole.

Plus shutting down things is easy as well. Things don't workout? Good news! You can shut down the infrastructure that very day instead of having to worry about the capital expenditure spent to build infrastructure, and without having to worry about its use later.

Simply put, AWS is infrastructure you can hire and fire at will.

[Ancapistani]

As a Staff+ Engineer -- Principal in a couple of weeks -- for me it's about "batteries included", support availability, and ability to hire.

Using a cloud platform means that while your needs are small, you're overpaying. Where it pays off is when you have a new requirement that needs to be met quickly.

I've done my share of managing database instances in the past. I can spin up a new RDS Postgres instance in much less time than I can configure one from scratch, though. Do we need a read replica? Multi-site failover? Do we need to connect it to Okta, or Formal, so we can stand up a process to provision access to specific databases, tables, or even columns? All of those things I can do significantly faster and more quickly on AWS than I can do it by hand.

What if a NoSQL database is the right solution for us? I have much less experience adminning those, so will either have to allocate a fair amount of my time to skill up or hire someone who already has those skills.

Need a scheduled task? Sure, I could set up a Jenkins server somewhere and we could use that... or we could just add an ECS scheduled task to our existing cluster.

Need an API endpoint to handle inbound Zoom events and forward them to an internal queue? Sure, I can set up a new VPC for that... that'll be a couple of days... or we whip up a Lambda, hook it up to API Gateway, and be up and running in a couple of hours.

AWS helps me do more in less time - and my time is a cost to the business. It's also extremely flexible, and will let us add things far more quickly than we otherwise could.

IMO, the correct comparison isn't "what would it cost to run this platform on Hetzner?" - it's "What would it cost to run it, plus what would cost to acquire the talent to build it, plus retain that talent to maintain it?"

AWS isn't competing with other infrastructure providers. They're competing with other providers and the salaries of the engineers you need to make them work.

[peanut-walrus]

It makes sense to think about price in the context of your business. If your entire infra cost is a rounding error on your balance sheet, of course you would pick the provider with the best features and availability guarantees (choose IBM/AWS). If your infra cost makes up a significant percentage of your operating expenses, you will start spending engineering effort to lower the cost.

That's why AWS can get away with charging the prices they do, even though it is expensive, for most companies it is not expensive enough to make it worth their while to look for cheaper alternatives.

[oersted]

It’s often less about engineering effort and more about taking some small risks to try less mainstream (but still relatively mature) alternatives by reasoning from first principles and doing a bit of homework.

From our experience, you can actually end up in a situation that requires less engineering effort and be more stable, while saving on costs, if you dare to go to a bit lower abstraction layers. Sometimes being closer to the metal is simpler, not more complex. And in practice complexity is much more often the cause of outages rather than hardware reliability.