[makkesk8]

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs"

This is a recurring topic here on HN and it boggles me and makes me wonder if people know that there are other platforms than aws, azure and google cloud out there that are very capable and much much cheaper.

Unless any of the big 3 has a feature or certification you need I don't see any reason to use them at all due to the insane complexity and cost.

So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point?

[CSDude]

Always this poor argument pops up. 1st AWS and big 3 is much more reliable. 2nd AWS is not only a VPS provider. Changing vendors is not the answer to all cost related problems.

[ethanwillis]

It's less reliable than a guy running a website on a raspberry pi with a solar panel from harbor freight http://pi.qcontinuum.com/

[qeternity]

This is not true at all. We run a large setup in OVH Gravelines and another in us-east-1 and we’ve not had a single outage in 3+ years with OVH. I can’t say the same for AWS.

[quicksilver03]

You have been lucky with Gravelines, OVH had major outages lasting several hours in 2017 (Roubaix, total loss of routing for all of the 6 DCs) and in one case close to 24 hours (Strasbourg, cascade of events resulting in total power failure http://status.ovh.net/?do=details&id=15162 ).

In that period (2015-2018) I used to run a fairly well known French website on OVH, and their network was very unstable, from equipment failures to way too many fat-fingering of routes. If you're able to easily switch traffic between OVH and AWS you're in a far better position than most people.

[qeternity]

We stayed away from Roubaix given the experimental design. Not sure why anyone would put production stuff there. I can’t speak to Strasbourg that sounds horrendous. But Gravelines has been rock solid and 4ms latency to London.

But at the end of the day outages happen everywhere, including AWS. We also have some kit at Hetzner and I think that a redundant setup across OVH and Hetzner will be a fraction of a cost of single AZ setup in AWS and yield far greater uptime.

We’ve commoditized the servers and services (cattle vs pets)...why would we treat the providers any differently? Use cheap components and lots of redundancy.

[gridlockd]

> 1st AWS and big 3 is much more reliable.

I think that's a myth. People assume it's true because it should be true. I don't think it is true.

> 2nd AWS is not only a VPS provider.

It's a glorified VPS provider. Most of the stuff doesn't matter to most of the people using AWS, but they go for it so they can put it on their resume and because they don't want to get fired for choosing something that's not a big name.

[jungturk]

EC2 might be a glorified VPS provider, but you seem to be ignoring the vast array of managed services in modern cloud providers (or unaware of their utility beyond padding resumes).

Load balancing, fault tolerance, high availability, arbitrary scale, messaging, orchestration, autoscaling, warehousing, big data processing, identity management, desktop management, secrets management, container registries, source code management, build tools, hardware test suites, gpu hardware, observability tools...

Those of use that use cloud providers know full well why we use them (and certainly know when not to).

[gridlockd]

If you really need any of that stuff you probably shouldn't use Amazon's managed version of it.

[rospaya]

Why? I run both on prem and cloud workloads (on more than one provider) so I'm wondering what's wrong with Amazon's managed services?

[MaxBarraclough]

> People assume it's true because it should be true. I don't think it is true.

It makes sense that bargain-bin providers would offer inferior reliability, but there are providers out there other than the big 3 cloud providers and the bargain-bin VPS providers.

GitHub for instance is apparently [0] hosted by Carpathia [1].

[0] https://github.com/holman/ama/issues/553

[1] http://www.carpathia.com/ (they should really fix https://www.carpathia.com )

[Pfhreak]

There should be data available for #1 -- it's not something we should need much subjective discussion around.

Re #2: I love all the other stuff. I never use EC2. Lambda, Cognito, DynamoDB, S3, CloudFront, Route 53, and API Gateway are my default stack, managed by Cloudformation. Granted, I'm doing smaller projects, but the costs are minimal, the setup time is trivial, the documentation is excellent, everything is nicely compartmentalized and 'just works' together. And I only pay for the actual traffic to the site.

[parsnips]

This is the way. We've given multiple clients very similar stacks for "api layers" and it's by far the most cost effective way to do AWS from both productivity and opex point of view.

[jorams]

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Storage has been very reliable and fast.

[saddlerustle]

This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky.

Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.

[fulafel]

There's options outside SaaS object storage though, eg running your own MinIO or Ceph, + DAS (or SAN) at your colo. Yes it's some hassle but if you have significant expenses from SaaS it might be worth checking out.

[dijit]

AWS S3 buckets have a limit of 5gb/s access to a single bucket.

Unless that limitation has changed in the last couple of years; I can easily make a system beat that, if that's the requirement.. and ultimately it does come down to understanding requirements. :\

I think generally people forget that cloud is just computers too, it's really not anything special, and amazon/google/microsoft are solving the general case (and, doing so well, actually) but this comes at a high premium.

[duggan]

It's 25Gbps now[1]. I have a rule of thumb: if I haven't refreshed something I think I know about AWS for more than half a year, it's probably out of date.

[1]: https://aws.amazon.com/premiumsupport/knowledge-center/s3-ma...

[saddlerustle]

That the throughput available to a single EC2 instance. S3 as a whole will do hundreds of Gbps no problem.

[speedgoose]

That's not a very common use case though. Most companies don't need to DDOS their cloud provider.

[jrockway]

25Gbps is 250 cable Internet users downloading the release of your new software. What you call a DDOS is actually a woefully underwhelming instantaneous transfer rate.

[Epixors]

If you're distributing releases (or anything, really) users should not have to go directly to your object store...

[speedgoose]

You should put your objects storage behind a CDN if you expect that many downloads. That's much cheaper and faster.

[jrockway]

I run my own Docker registry for my personal projects backed by DO Spaces. It was so slow that I couldn't actually believe it, and had to instrument the registry code extensively to account for all the time spent in the storage API, to see what the actual problem was. Turns out Spaces is really, really slow.

[csunbird]

is it that bad? we are thinking about bootstrapping a project, which will use spaces for file storage and we will be using them frequently.

[barake]

Provisioning of new Spaces was recently disabled in some regions while they add capacity. Vaguely concerning IMO and has caused me to seriously reconsider using DO Spaces for a new product I'm building.

https://www.digitalocean.com/docs/release-notes/upcoming/spa...

[csunbird]

I do know that they have limited the creation and looking at the docs for limitation, it seems like they have a 750 requests per second limit per space, which might not scale well. My idea was to create multiple spaces to distribute a load, e.g. sharding per customer, but if the availability is not that good, I might have to re-consider our choice.

The main reason we wanted to work with DO is that the egress traffic cost is reasonable, unlike AWS or Google Cloud where it is questionably high.

[jorams]

It was that bad 2 years ago, in the AMS3 region. I don't know how it is these days, but I expect after 2 years it has matured at least a little bit. They also now include a CDN, which might improve things a bit if it reduces the load on the buckets behind it.

Honestly it soured my opinion on DO a bit. I'd previously had a droplet there for several years, which never seemed to experience any issues at all.

[csunbird]

In our case, we can not use the CDN, unfortunately, because the objects change a lot.

[jonatron]

I tried it for a small personal project, just for somewhere to put some small images. On pages of about 30 images, usually one of them just wouldn't load. It didn't look reliable to me either.

[l33tman]

I'm sure there are other services that are cheaper in some of the areas, but at least the argument about "insane complexity" is moot; if you've used AWS before it's no problem to set something up from scratch. I've done it numerous times and it's a one day job to setup most of the infrastructure required with VPCs, IAM users, policies, databases, scaling groups, storage configs etc.. That part of it is not really an argument to switch at least.

[PetahNZ]

Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?

[dijit]

The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware.

I'm not sure where the idea came from that this is hard to do.

To answer your questions in order:

1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy.

I'm not saying it's black and white; but I hope that you got your velocity required to hit market faster and have made more than you spent because this is the price you paid; and now to get out you'll have to invest a little time on cleaning house, that's the reality of lock-in.

2) S3 is pretty easy as there are "s3 compatible" FOSS projects; min.io comes to mind, or ceph with a RADOS plugin, or Riak with the s3 plugin... there's also s3proxy with a multitude of backends..

3) Aurora serverless is replaced by knative

4) Beanstalk is just classical servers with an auto-scaler component, auto-scaling depends greatly on your provider, so I can't say how easy or hard it will be, if you're using kubernetes then understanding your load should be easy at least.

[marcinzm]

Now you have five more things you need to research, learn, keep track of, debug, etc. Plus, in my experience, anyone who says kubernetes is easy to run (cloud or native) has never had to run kubernetes in production.

[dijit]

Sure, then pay?

I don't understand the argument: "I want to outsource understanding but I want to save costs";

You can think of things as a spider-graph of three points:

Quality -- Low-Cost -- Knowledge-Required.

No solution can score high points on all three reasonably.

Anyway, things are a bit skewed because I'm an infrastructure type, and people in my profession really do think of systems administration tasks as being "very easy" and if done right soak up nearly no time at all, but developers don't like hearing that because sysadmins are "old world".

I don't really care if you're paying someone elses sysadmins or not, the fact remains that you're going to be spending something in that area, and if you balk at the cost of cloud then maybe taking ownership of what they do can help optimise costs.

Obviously they put a premium on their own time in these areas.

[aripickar]

I disagree. Once you increase your knowledge of AWS and associated systems, you can decrease cost of what you are doing through tips like what you see in this article. I don't quite get the point about optimizing cloud costs solely by switching to self hosted? Like of course you could do that, but you could also optimize costs by doing what the article says.

Full Disclosure: I work at AWS building tools to help customers do cost optimization.

[dijit]

Someone made a good reply to this and as I was replying to that they deleted it, so I'll copy it here:

----

> Have you actually done a cost-benefit analysis on some of these solutions?

Yes, I even gave a talk at google in stockholm about it.

For my use-case, hybrid was best, with no cloud lock-in aside from Google Storage Buckets (which can be replaced) but I went into detail about that in the talk.

> Take your Riak / S3 plugin. What do your servers cost to run that cluster?

Depends a lot, don't you think?

> How much time do you spend managing it?

Depends again, if it's anything like my elasticseach clusters then about 2-man hrs/mo.

> How do you test your backups?

Continuously, and with alerting.. and, you should be doing this anyway.

> Are you going to target the same SLOs for durability that S3 offers?

Depends on the business, the whole point of SLO is that you pay in what it's worth to the business.

> Do you run multi-data center for high availability?

Depends on SLO.

> In many cases the cheaper or self-hosted solutions have costs that you aren't accounting for.

Yes, physical machines often need some hand-holding, VPS's can have brown outs, but this is true in AWS's EC2 anyway.

Ultimately, this is where the cost increase will be.. but defining it is important, I've deployed cloud and physical (as stated) and it's true that physical machines are not as problem-free as our GCE ones- but we pay about 50% less than the GCE equiv instances, so it's "worth" spending time automating the unpredictable.

> Sometimes that's fine, but "just run it yourself" is as worthless as saying "just ship it to AWS" unless you actually think through the impact.

This is kind of the main point I always make.. understand your trade-offs, don't buy into proprietary tech. Cloud is a fantastic way to prototype and bootstrap but it's /usually/ better to have a migration plan to optimise costs in the future.

If you fail to take that into account then I don't have sympathy for you, because you put the project at risk. Financial in-viability is a risk.

[owenmarshall]

That was me ;) I deleted it because I read your second post and realized I'd completely misread you and actually agree almost in entirety.

> understand your trade-offs, don't buy into proprietary tech. Cloud is a fantastic way to prototype and bootstrap but it's /usually/ better to have a migration plan to optimise costs in the future.

Preach.

[pm90]

I don't think moving cloud providers is an effective way to solve for cost problems; especially considering the engineering cost involved in making these migrations, which reduce the marginal utility of such a migration to 0, possibly even <0.

If you're really looking to save on costs, hosting solutions (i.e. hosting racks or some managed solution) are probably what you would need to look at. But then there are other costs involved there as well (infrastructure team, upfront capex costs etc.), but it might still be worth it if you run e.g. a ton of batch processing over a ton of data.

[marcinzm]

>So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point?

Because it takes X hours of effort to cut costs and using another provider would have required Y hours of effort where Y >> X. Y may be greater due to reliability issues, missing features that you need to build yourself, training costs of new employees, etc.

edit: Also X is paid once you're succeeding, Y has to be paid before you're succeeding which makes Y even more expensive in opportunity cost.

[yellow_mixer]

It’s also much easier to hire engineers to develop and maintain theses platforms.

[secondcoming]

It's probably because when devs jump into the cloud they can be distracted by Shiny New Toys and cost is only considered once the first bill comes in.