[motorest]

> I'd rather go with Hetzner and cobble up together some failover than pay AWS extortion.

Comments like this are so exaggerated that they risk moving the goodwill needle back to where it was before. Hetzner offers no service that is similar to DynamoDB, IAM or Lambda. If you are going to praise Hetzner as a valid alternative during a DynamoDB outage caused by DNS configuration, you would need to a) argue that Hetzner is a better option regarding DNS outages, b) Hetzner is a preferable option for those who use serverless offers.

I say this as a long-time Hetzner user. Herzner is indeed cheaper, but don't pretend that Herzner let's you click your way into a highly-availale nosql data store. You need non-trivial levels of you're ow work to develop, deploy, and maintain such a service.

[1dom]

> but don't pretend that Herzner let's you click your way into a highly-availale nosql data store.

The idea you can click your way to a highly available, production configured anything in AWS - especially involving Dynamo, IAM and Lambda - is something I've only heard from people who've done AWS quickstarts but never run anything at scale in AWS.

Of course nobody else offers AWS products, but people use AWS for their solutions to compute problems and it can be easy to forget virtually all other providers offer solutions to all the same problems.

[ApolloFortyNine]

>The idea you can click your way to a highly available, production configured anything in AWS - especially involving Dynamo, IAM and Lambda

With some services I'd agree with you, but DynamoDB and Lambda are easily two of their 'simplest' to configure and understand services, and two of the ones that scale the easiest. IAM roles can be decently complicated, but that's really up to the user. If it's just 'let the Lambda talk to the table' it's simple enough.

S3/SQS/Lambda/DynamoDB are the services that I'd consider the 'barebones' of the cloud. If you don't have all those, you're not a cloud provider, your just another server vendor.

[manquer]

> Lambda are easily two of their 'simplest'

Not if you want to build something production ready. Even a simple thing like say static IP ingress for the Lambda is very complicated. The only AWS way you can do this is by using Global Accelerator -> Application Load Balancer -> VPC Endpoint -> API Gateway -> Lambda !!.

There are so many limits for everything that is very hard to run production workloads without painful time wasted in re-architecture around them and the support teams are close to useless to raise any limits.

Just in the last few months, I have hit limits on CloudFormation stack size, ALB rules, API gateway custom domains, Parameter Store size limits and on and on.

That is not even touching on the laughably basic tooling both SAM and CDK provides for local development if you want to work with Lambda.

Sure Firecracker is great, and the cold starts are not bad, and there isn't anybody even close on the cloud. Azure functions is unspeakably horrible, Cloud Run is just meh. Most Open Source stacks are either super complex like knative or just quite hard to get the same cold start performance.

We are stuck with AWS Lambda with nothing better yes, but oh so many times I have come close to just giving up and migrate to knative despite the complexity and performance hit.

[ApolloFortyNine]

>Not if you want to build something production ready.

>>Gives a specific edge case about static IPs and doing a serverless API backed by lambda.

The most naive solution you'd do on any non-cloud vendor, just have a proxy with a static ip that then routes traffic whereever it needed to go, would also work on AWS.

So if you think AWS's solution sucks why not just go with that? What you described doesn't even sound complicated when you think of the networking magic behind the scenes that will take place if you ever do scale to 1 million tps.

[manquer]

> Production ready

Don’t know what you think should mean but for me that means

1. Declarative IaaC in either in CF/terraform

2. Fully Automated discovery which can achieve RTO/RPO objectives

3. Be able to Blue/Green and % or other rollouts

Sure I can write ansible scripts, have custom EC2 images run HA proxy and multiple nginx load balancers in HA as you suggest, or host all that to EKS or a dozen other “easier” solutions

At the point why bother with Lambda ? What is the point of being cloud native and serverless if you have to literally put few VMs/pod in front and handle all traffic ? Might as well host the app runtime too .

> doesn’t even sound complicated .

Because you need a full time resource who is AWS architect and keeps up with release notes and documentation or training and constantly works to scale your application - because every single component has a dozen quotas /limits and you will hit them - it is complicated.

If you spend few million a year on AWS then spending 300k on an engineer to do just do AWS is perhaps feasible .

If you spend few hundred thousands on AWS as part of mix of workloads it is not easy or simple.

The engineering of AWS impressive as it maybe has nothing to the products being offered . There is a reason why Pulumi, SST or AWS SAM itself exist .

Sadly SAM is so limited I had to rewrite everything to CDK in couple of months . CDK is better but I am finding that I have to monkey patching limits on CDK with the SDK code now, while possible , the SDK code will not generate Cloudformation templates .

[motorest]

> Don’t know what you think should mean but for me that means

I think your inexperience is showing, if that's what you try to mean by "production-ready". You're making a storm in a teacup over features that you automatically onboard if you go through an intro tutorial, and "production-ready" typically means way more than a basic run-of-the-mill CICD pipeline.

As most of the times, the most vocal online criticism comes from those who have the least knowledge and experience over the topic they are railing against, and their complains mainly boil down to criticising their own inexperience and ignorance. There is plenty of things to criticize AWS for, such as cost and vendor lock-in, but being unable and unwilling to learn how to use basic services is not it.

[motorest]

> Even a simple thing like say static IP ingress for the Lambda is very complicated.

Explain exactly what scenario you believe requires you to provide a lambda behind a static IP.

In the meantime, I recommend you learn how to invoke a lambda, because static IPs is something that is extremely hard to justify.

[manquer]

Try telling that to customers who can only do outbound API calls to whitelisted IP addresses

When you are working with enterprise customers or integration partners it doesn’t even have to be regulated sectors like finance or healthcare, these are basic asks you cannot get away from .

people want to be able to know whitelist your egress and ingress IPs or pin certificates. It is not up to me to say on efficacy of these rules .

I don’t make the rules of the infosec world , I just follow them.

[motorest]

> Try telling that to customers who can only do outbound API calls to whitelisted IP addresses

Alright, if that's what you're going with then you can just follow a AWS tutorial:

https://docs.aws.amazon.com/lambda/latest/dg/configuration-v...

Provision an elastic IP to have your static IP address, set the NAT gateway to handle traffic, and plugin the lambda to the NAT gateway.

Do you think this qualifies as very complicated?

[1dom]

> With some services I'd agree with you, but DynamoDB and Lambda are easily two of their 'simplest' to configure and understand services, and two of the ones that scale the easiest. IAM roles can be decently complicated, but that's really up to the user. If it's just 'let the Lambda talk to the table' it's simple enough.

We agree, but also, I feel like you're missing my point: "let the Lambda talk to the table" is what quickstarts produce. To make a lambda talk to a table at scale in production, you'll want to setup your alerting and monitoring to notify you when you're getting close to your service limits.

If you're not hitting service limits/quotas, you're not running even close to running at scale.

[motorest]

> The idea you can click your way to a highly available, production configured anything in AWS - especially involving Dynamo, IAM and Lambda - is something I've only heard from people who've done AWS quickstarts but never run anything at scale in AWS.

I'll bite. Explain exactly what work you think you need to do to get your pick of service running on Hetnzer to have equivalent fault-tolerance to, say, a DynamoDB Global Table created with the defaults.

[esskay]

Are you Netflix? Because is not theres a 99% probability you dont need any of those AWS services and just have a severe case of shiny object syndrome in your organisation.

Plenty of heavy traffic, high redundancy applications exist without the need for AWS (or any other cloud providers) overpriced "bespoke" systems.

[rafaelmn]

To be honest I don't trust myself running a HA PostgreSQL setup with correct backups without spending an exorbitant effort to investigate everything (weeks/months) - do you ? I'm not even sure what effort that would take. I can't remember last time I worked with unmanaged DB in prod where I did not have a dedicated DBA/sysadmin. And I've been doing this for 15 years now. AFAIK Hetzner offers no managed database solution. I know they offer some load balancer so there's that at least.

At some point in the scaling journey bare metal might be the right choice, but I get the feeling a lot of people here trivialize it.

[cdelsolar]

If youre not Netflix then just sudo yum install postgresql and pg_dump every day, upload to S3. Has worked for me for 20 years at various companies, side projects, startups …

[motorest]

> If youre not Netflix then just sudo yum install postgresql and pg_dump every day, upload to S3.

database services such as DynamoDB support a few backup strategies out of the box, including continuous backups. You just need to flips switch and never bother about it again.

> Has worked for me for 20 years at various companies, side projects, startups …

That's perfectly fine. There are still developers who don't even use version control at all. Some old habits die hard, even when the whole world moved on.

[rafaelmn]

What happens when the server goes down ? How do you update it ?

[cdelsolar]

you stand up another db server and load the last good dump into it i suppose

[esskay]

If it requires weeks/months to sort setting that up and backups then you need a news ops person as that's insane.

If you're doing it yourself, learn Ansible, you'll do it once and be set forever.

You do not need "managed" database services. A managed database is no different from apt install postgesql followed by a scheduled backup.

It genuinely is trivial, people seem to have this impression theres some sort of unique special sauce going on at AWS when there really isn't.

[jameshart]

That doesn’t give you high availability; it doesn’t give you monitoring and alerting; it doesn’t give you hardware failure detection and replacement; it doesn’t solve access control or networking…

Managed databases are a lot more than apt install postgresql.

[byryan]

If you're doing it yourself, learn Ansible, you'll do it once and be set forever.

You do not need "managed" database services. A managed database is no different from apt install postgesql followed by a scheduled backup.

Genuinely no disrespect, but these statements really make it seem like you have limited experience building an HA scalable system. And no, you don't need to be Netflix or Amazon to build software at scale, or require high availability.

[bostik]

Backups with wall-g and recurring pg_dump are indeed trivial. (Modulo S3 outage taking so long that your WAL files fill up the disk and you corrupt the entire database.)

It's the HA part, especially with a high-volume DB that's challenging.

[rafaelmn]

But that's the thing - if I have an ops guy who can cover this then sure it makes sense - but who does at an early stage ? As a semi competent dev I can setup a terraform infra and be relatively safe with RDS. I could maybe figure out how to do it on my own in some time - but I don't know what I don't know - and I don't want to do a weekend production DB outage debugging because I messed up the replication setup or something. Maybe I'm getting old but I just don't have the energy to deal with that :)

[motorest]

From your comment, you don't even have the faintest idea of what is the problem domain. No wonder you think you know better.

[motorest]

> Are you Netflix? Because is not theres a 99% probability you dont need any of those AWS services and just have a severe case of shiny object syndrome in your organisation.

I think you don't even understand the issue you are commenting on. It's irrelevant if you are Netflix or some guy playing with a tutorial. One of the key traits of serverless offerings is how it eliminates the need to manage and maintain a service or even worry about you have enough computational resources. You click a button to provision everything, you configure your clients to consume that service, and you are done.

If you stop to think about the amount of work you need to invest to even arrive at a point where you can actually point a client at a service, you'll be looking at what the value of serverless offerings.

Ironically, it's the likes of Netflix who can put together a case against using serverless offerings. They can afford to have their own teams managing their own platform services with the service levels they are willing to afford. For everyone else, unless you are in the business of managing and tuning databases or you are heavily motivated to save pocket change on a cloud provider bill, the decision process is neither that clear not favours running your own services.

[sofixa]

> Plenty of heavy traffic, high redundancy applications exist without the need for AWS (or any other cloud providers) overpriced "bespoke" systems.

And almost all of them need a database, a load balancer, maybe some sort of cache. AWS has got you covered.

Maybe some of them need some async periodic reporting tasks. Or to store massive files or datasets and do analysis on them. Or transcode video. Or transform images. Or run another type of database for a third party piece of software. Or run a queue for something. Or capture logs or metrics.

And on and on and and on. AWS has got you covered.

This is Excel all over again. "Excel is too complex and has too many features, nobody needs more than 20% of Excel. It's just that everyone needs a different 20%".

[esskay]

You're right AWS does have you covered. But that doesn't mean thats the only way of doing it. Load balancing is insanely easy to do yourself, databases even easier. Caching, ditto.

I think a few people who claim to be in devops could do with learning the basics about how things like Ansible can help them as there's a fair few people who seem to be under the impression AWS is the only, and the best option, which unless you're FAANG really is rarely the case.

[sofixa]

You can spin up a redundant database setup with backups and monitoring and automatic fail over in 10 mins (the time it takes in AWS)? And maintain it? If you've done this a few times before and have it highly automated, sure. But let's not pretend it's "even easier" than "insanely easy".

Load balancing is trivial unless you get into global multicast LBs, but AWS have you covered there too.

[ta1243]

You could never run a site like hacker news on a single box somewhere with a backup box a couple of states away.

(/s, obviously)

[sofixa]

And have the two fail at the same time because similarly old hardware with similarly old and used disks fails at roughly the same time :)

[motorest]

> You're right AWS does have you covered. But that doesn't mean thats the only way of doing it. Load balancing is insanely easy to do yourself, databases even easier. Caching, ditto

I think you don't understand the scenario you are commenting on. I'll explain why.

It's irrelevant if you believe that you are able to imagine another way to do something, and that you believe it's "insanely easy" to do those yourself. What matters is that others can do that assessment themselves, and what you are failing to understand is that when they do so, their conclusion is that the easiest way by far to deploy and maintain those services is AWS.

And it isn't even close.

You mention load balancing and caching. The likes of AWS allows you to setup a global deployment of those services with a couple of clicks. In AWS it's a basic configuration change. And if you don't want it, you just tear down everything with a couple of clicks as well.

Why do you think a third of all the internet runs on AWS? Do you think every single cloud engineer in the world is unable to exercise any form of critical thinking? Do you think there's a conspiracy out there to force AWS to rule the world?

[ViewTrick1002]

If you need the absolutely stupid scale DynamoDB enables what is the difference compared to running for example FoundationDb on your own using Hetzner?

You will in both cases need specialized people.

[mschuster91]

> Hetzner offers no service that is similar to DynamoDB, IAM or Lambda.

The key thing you should ask yourself: do you need DynamoDB or Lambda? Like "need need" or "my resume needs Lambda".

[motorest]

> The key thing you should ask yourself: do you need DynamoDB or Lambda? Like "need need" or "my resume needs Lambda".

If you read the message you're replying to, you will notice that I singled out IAM, Lambda, and DynamoDB because those services were affected by the outage.

If Hetzner is pushed as a better or even relevant alternative, you need to be able to explain exactly what you are hoping to say to Lambda/IAM/DynamoDB users to convince them that they would do better if they used Hetzner instead.

Making up conspiracy theories over CVs doesn't cut it. Either you know anything about the topic and you actually are able to support this idea, or you're an eternal September admission whose only contribution is noise and memes.

What is it?

[darkwater]

Well, Lambda scales down to 0 so I don't have to pay for the expensive EC2 instan... oh, wait!

[sgarland]

> click your way into a HA NoSQL data store

Maybe not click, but Scylla’s install script [0] doesn’t seem overly complicated.

0: https://docs.scylladb.com/manual/stable/getting-started/inst...