[jitl]

We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client.

In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand damage alone. Possibly even company ending.

In such a situation, failure is not an option, and you must pick the least risky solution. Moving to an unmanaged cluster system and figuring out your own point-in-time backup/restore, access control provisioning, etc etc has a lot more unknown unknowns than sticking with the managed database vendor you know. The potential failure scenarios of Citus have scary worst cases - we get backup and restore wrong but it seems to work fine in test, then we move to Citus, then something breaks and we can’t restore from backup after all. It’s equally bad to mis-estimate the amount of time needed to bring up the new system. Let’s say you estimate 6 months to get parity with RDS built in features needed to survive disaster and start moving data over, but instead it takes 10 months. Is there enough time left to finish before going hard down? The clock is ticking. Staying with RDS keeps a whole class of new risk out of the picture.

At least here at Notion, NO ONE wanted to build something complicated for fun. We really wanted the company we’d spent years working for and on-call for to survive.

Our story: https://www.notion.so/blog/sharding-postgres-at-notion

[hobs]

Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.

[jitl]

It's not just manage Postgres, it's manage a Citus cluster - (unmanaged Postgres + postgres experts + time for them to implement their stuff) just gets us to parity with RDS but doesn't solve our sharding problem. We asked our Postgres consultants & networks to see if we could find Citus experts we could bring on full-time but didn't have great success. Most of the experts we talked to suggested application level sharding, and it seems like it worked out okay.

[hobs]

Absolutely, I am just saying that you are talking on now all the inconsistencies of a third party management system and building your solution on top of that; you don't get the infra savings and benefits of managing your own, you gain some velocity for now and as big name clients probably will be stable for a few years.

I had a problem just recently where I worked at a place that's using blue/green aws rds deployments with mysql replication, and binlogs cant be moved in that service.

This is something that is bog simple in a non-managed service, and as a result we can either manage app replication, re-sync data on each b/g upgrade, or do physical replication (slow). My point isn't that rds is bad, it's just that if you are already deciding to implement your own significant infrastructure on top of database it seems weird to me to not just have the knobs on the thing itself.

Though you could say the same is true of the storage, and tbqh most of the cloud storage is dogshit these days but we just deal with it.

[vbezhenar]

It seems that this day the art of configuring a database has been long lost. I also completely don't understand the issue. Just buy two huge behemoth servers, put your postgres there in a replicated mode and move on. It'll sustain huge load. Surely those companies can afford to hire one sysadmin.

[jitl]

You can’t necessarily play Cookie Clicker with database hardware scaling and have a good time. Query performance and upkeep processes often begin to degrade well before a table reaches the maximum hardware-bound size. We were using an instance with 96 cores and 350gb of RAM which seems over provisioned on paper and still hitting a variety of issues like stalled out Postgres auto-vacuum.