[Jonnax]

At what scale would you want to use RDS rather than using an EC2 instance with Postgres installed?

Assuming that the operator has the skills to manage Postgres.

It's not like RDS does something complex like Geodistribution, right?

Also what is the scaling like? Is it automatic? How quickly can you handle more connections? Because my understanding was that it was slow.

I did have a play with their RDS Postgres nonths before, and I managed somehow to crash it requiring a restore from snapshot. Also their smallest instance was quite expensive for the performance.

[makmanalp]

I'd ask the opposite question - at what scale would you want to have your own custom setup rather than RDS? Managing your own database infrastructure for workloads other than "a few queries a second" is hard work with a lot of pitfalls, and you better be at a size that there's some benefit (high levels of customization, use case specific tuning, economies of scale, etc). As a person who does exactly this for a living, I'd rather shell out for RDS or a similar offering than my own setup most of the time. Especially at first, before you discover what exactly you /don't/ like about it or what you'd want different.

[tmpz22]

Is it hard work though? In a couple hours you should be able to setup automatic backups and practice going through the recover process a couple times. That's all there is for most small-business setups, but if you are daring you can now do whatever you want with the config file, install extensions, setup basic system monitoring (CPU/Ram usage, disk usage, etc.). GCP/Digital Ocean let you look at node resource usage automatically, and since Postgres is probably the only process it means you don't even need to set that up!

[malisper]

> In a couple hours you should be able to setup automatic backups and practice going through the recover process a couple times.

Unfortunately there's a lot more too it than that. You need to handle when the backup job fails or dies, have a process for deleting old backups, etc. Not just that, but if you have multiple Postgres instances, you need to do this work for each machine. I've seen first hand this kind of stuff become a huge distraction. It's often worth it to pay AWS a bit more in exchange to not worry about this stuff.

[tmpz22]

> Unfortunately there's a lot more too it than that.

Is there though? Consider what I would argue to be the "average" case:

* Your database never exceeds > 40% resource usage

* You service fewer then 1m queries/day

* You never burst more then 1k queries/minute

* You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails

* You have a script tied to a cronjob which deletes old backups, with basic error handling that sends you a Slack DM if it fails

What percentage of companies need more then that?

[fragmede]

> * Your database never exceeds > 40% resource usage > * You service fewer then 1m queries/day > * You never burst more then 1k queries/minute

How do I know it doesn't exceed 40% usage? Better yet, who's holding the pager when it does? If/when it does, who's product launch is dead in the water while the db is reconfigured onto a larger instance? What product isn't being delivered because we're faffing about with the database instead of product code?

> * You have a script tied to a cronjob that backs up the database, with basic error handling that sends you a Slack DM if it fails > * You have a script tied to a cronjob which deletes old backups, with basic error handling that sends you a Slack DM if it fails

Who's responsible for restoring from backup every week/month/quarter, to assert they actually work, with whatever changes have been made recently? Untested backups are Shrodingers backups.

Just how well tested is this script? Does it properly error out if the script fails to be run? What if a firewall rule accidentally gets set that blocks egress from the backup box to the Internet (for security); who/how/what gets notified instead? Who's deliverables are slipping because the backups randomly stopped working?

> What percentage of companies need more then that?

That's a fair question, but Amazon's done far more research than I, possibly you on that topic. The real question is, of companies that don't need more than that, how many companies want to hire somebody to take on those responsibilities part-time? How many companies have the expertise to even hire somebody qualified to do that part-time? And since those people are managing the DB part time, how many of them are giving it the attention it needs, and aren't distracted by other responsibilities to the company?

None of those problems are insurmountable, but they're far from most business' core competency, and time I'm spending dealing with postgresql.conf (or my.cnf) is time I'm not dealing with other issues. Don't get me wrong, there's still a time and place for managing database instances, but IMO small business (small > tiny) aren't the appropriate place for that. I'd be interested in hearing if someone's run the numbers to justify it though! (Especially if it falls in favor of running it yourself.)

[sciurus]

Any that can't afford more than a couple minutes of downtime when a server fails.

[viraptor]

That's definitely not an "average" company. It's also a really small number of companies that really can't afford that, rather than "earn less money than usual".

[forkLding]

I used both too and I think it really depends if you want to pay for it. For small to medium independent projects, I think a EC2 instance that sometimes cuts past that free tier usage is fine, otherwise RDS can be overkill and can seriously eat into costs. I would say a beginner doing independent projects should strongly consider EC2 instances instead of RDS.

[reilly3000]

RDS can scale read replicas and fail over to master, but aurora and it’s serverless option is much better for auto-scaling. Behind the scenes storage is decoupled from compute with makes scaling fundamentally easier.

EC2 is your only choice if you want a database that AWS doesn’t support, such as Rethink or Cassandra (they just recently launched a managed Cassandra service though). EC2 is also your only choice if you need full control of the DB, such as using many Postgres extensions and foreign data wrappers. Even some triggers and UDFs are limited.

A self-managed, auto-scaling, cross-AZ replicated DB setup is no small matter with EC2. Not to mention logging, metrics, patching of the DB and underlying OS. It’s 100% doable, but one should only proceed with that course with understanding of the human costs.

Personally, I’ve been choosing FaunaDB these days when possible. It’s a no-ops managed service and has on-prem/VPC options. I just write graphQL clients and move on with my life, the rest just works.

[CoolGuySteve]

I went with Aurora Postgres recently for a new application and to be honest it fucking sucks.

When it scales up, queries start to fail and there are long delays. I had to add a try/retry loop around all my inserts to avoid losing data.

I definitely wouldn't recommend it for bursty applications where it needs to rapidly scale up.

The thing that bugs me is that I know a similar workload works fine on a relatively cheap c3.xlarge so I'm not actually saving much money with Aurora.

[nostrebored]

... were you not handling errors on database inserts through well established concepts like exponential backoff and retry already?

that seems like a colossally bad idea

If you don't need the striped storage model and guarantees that Aurora provides, it might not be the service for you

[CoolGuySteve]

Normally I’d agree with you but I was only doing a dozen or so inserts per second for maybe a minute or two after hours of idleness.

There was no back off because it was never needed before, even the shittiest instance could handle it without issue but Aurora failed spectacularly.

[meritt]

RDS (postgres, mysql, maria) is basically just a managed ec2 instance. The instance cost is about ~2x the comparable price for an ec2 instance, which seems reasonable to me. Storage costs are pretty comparable to EBS. You can do push-button upgrades to increase the capacity, but it's slow. The main benefits I think you get over self-managed is automated / on-demand backups, pretty seamless software upgrades/patches, you can quickly spin-up a duplicate instance from a snapshot for testing/distributing workloads/etc, and replication works pretty seamlessly.

Provisioned IOPS is one area that can get expensive very quick, but people often don't realize that you get 3 PIOPS included with every 1GB of allocated storage, so you really don't need to pay for provisioned IO if you have a decent amount of storage.

If you want auto-scaling you need to look at Aurora or Redshift, which are quite different and significantly more expensive. I've not used those.

[zedpm]

It isn't a matter of getting to a certain scale before you use RDS, if that's what you're asking. The value of RDS is streamlining and automating db administrative tasks. If you want to update from Postgres 11.5 to 11.6, for example, you just change that setting on your RDS instance and it happens, either immediately or at some scheduled maintenance window. I f you want a hot standby in a separate availability zone, it's trivial to add one. Read replica? Trivial to add.

In general, I've found it makes sense to pay the premium for RDS and spend my and my team's time on more valuable work than db admin tasks.

[malisper]

I've previously administrated a cluster Postgres instances with a total of about 1PB of data. My recommendation is that you should use RDS unless you have a reason not to.

RDS takes care of tons of administrative tasks such as backups, replication, failover, and database upgrades. Yes you can setup backups yourself, but the on going maintenance is going to be a pain. You need to deal with what happens when a backup fails, have a playbook for restoring from backup, cleanup your old backups, etc. These are tasks that are extremely dangerous if you get something wrong and they are completely taking care of for you by RDS.

[mdeeks]

I'll take a shot at this. There is always an asterisk under every one of these. Every company and situation is different.

If you're a tiny startup or hobby with literally no money, it might make sense for you to manage it yourself because you have no choice.

Once you have some money and a viable business, then your value is no longer your ability to spend your time running Postgres, ensuring backups and restores work, creating replicas, upgrading software, and setting up all of the monitoring tools. You provide MUCH more value spending your time and abilities building things that are core to the business that let you make money and grow.

No doubt you can do it all and save some cash. But you have to do it regularly if you want confidence that everything you have built still works. With RDS, you pay them some extra for a near guarantee that it will all just work 100% of the time.

Once you become a large company with tons of engineers and you start to bump into limits of RDS, then it might make sense to run it yourself again. It is a significant burden to do it correctly 100% of the time. Your entire business can fail if you don't do your job right.

[nunez]

If I were a "tiny startup", I would absolutely leverage RDS in the first instance. Last thing I want to spend time doing is setting us Postgres and accumulating tech debt from a bad or suboptimal install. There are RDS instances that qualify for Free Tier as well if you're running _really_ lean.

In fact, the only time that I would consider migrating to a DB on EC2 is if the database isn't supported by RDS (DB2, for example, though I think there are solutions for this now), the licensing for RDS makes it cost-prohibitive or if the client needed to be able to really tweak the database engine or daemons.

[mdeeks]

The key words were "literally no money". By tiny startup I meant one or two people without any funding, revenue, and saddled with life expenses like rent, food, college savings, etc.

I definitely agree that a DB on EC2 is tech debt. But sometimes that is something you have to accept for immediate financial reasons while you figure out how to create a product with revenue to cover costs.

[paxy]

RDS/Aurora does do autoscaling, backups, georeplication, encryption etc. It's more a matter of time & convenience rather than skill. Sure you could do all of it yourself on an EC2 instance, but at some point it becomes a big chunk of your job, and you would rather be spending your time on other things.

[nunez]

You would move to RDS the minute you know you're committed to AWS and know that you don't want to worry about things like backups, upgrades or clustering. The disadvantage to doing so is that you lose a lot of administrative privileges to the database server itself and you don't get access to the filesystem or underlying OS. We had trouble migrating a sizable SQL Server installation onto RDS because it had a ton of triggers and stored procedures (SPs) that relied on files in the filesystem.

[bradknowles]

Yeah, well. That’s SQL Server for you. There’s a lot of things that RDS can’t do for you in the SQL Server flavor.

RDS MySQL != RDS Postgres && RDS (MySQL || Postgres) >> RDS SQL Server

[pestaa]

There are advantages when using RDS other than scaling.

The performance dashboard is especially nice.

The reusable sets of configuration is convenient and the UI makes it easy to compare original vs. changed values.

The instance upgrade is not seamless, though, it is normally scheduled for the next maintenance window, unless you decide to apply and reboot immediately.

I agree the service is expensive, but setting up a db server for production takes a lot of time and expertise.

[pdeva1]

one thing that doesn't get talked about RDS is that network cost for replication of data for RDS multi A-Z deployments is free. Depending on how much you write to RDS, this cost can dominate cpu/memory costs on non-RDS installations.

[encoderer]

For Cronitor, we did it when we needed replication.