[zie]

I would argue, you handle the hyper-scale use case when you are actually in hyper-scale. Trying to pre-maturely optimize this is almost always a waste of time and chances are you will screw it up anyway. Almost nobody gets to that scale anyway. If you do get to that scale, you have the money and resources to fix the problem(s) at that time.

[dalyons]

i mean, sort of? There is some subtly lost in this oft-repeated advice. i've worked at 3 companies now that were initially based on a single RDBMS but have outgrown the scale of what is reasonable to serve off that architecture. They are consumer scale (10s of mill) users, but not hyperscale (IMHO 100m+). The amount of engineering cost to migrate a complicated growing company/product off a mono-db architecture is astounding. Conservatively i'm talking 10+ dev years of effort, at each company. Easily 10s of millions of $$$, maybe 100m+. None of them are "finished". It's really really time consuming and hard, once you have 100s of tables, 100s of thousands of lines of code, dozens of teams, etc.

I'm all about avoiding premature optimization, and its fine to start with a classic postgres. But please don't cling to that - if you see MVP success and you actually have a reasonable chance of getting to >1mill users (ie, a successful B2C product) please please dont wait to refactor your datastore to a more scalable solution. You will pay dearly if you wait too long. Absolutist advice serves noone well here - it really does depend on what your goals are as a company.

[zie]

Of course subtlety matters, but as you start scaling and noticing pain points, that is when you start working towards fixing them. First you just throw hardware at the problem and that tends to scale really really well for a really long time. It's pretty rare, even at very large scale that you MUST move off of PG, there are plenty of well tested scaling solutions, if you have the $$$'s to spend.

10+ years of dev work for a few hundred tables worries me a lot. My last conversion was about 20 years of data across a few hundred tables and we did two-way data synchronization across DB products with about 1 month of work, with 2 devs. We kept the sync running for over a year in production because we didn't want to force users over to the new system in a big hurry. We only stopped because the license on the old DB product finally expired and nobody wanted to pay for it anymore.

[dalyons]

Yes again the common refrains - just throw hardware at it. I/we of course know this and all the systems I’m referring to did that first until they couldn’t. But you’re kind of missing my point - im saying by the time you are noticing scale pain points it’s often too late. Too late insofar as your system has likely grown so much in breadth (complexity, features, subsystems, lines of code, services, etc) that all depend on this one db. All this vast amount of stuff all written assuming all tables are accessible to everyone. It becomes a tangled web of data access patterns / tables that is very hard to break apart.

Nevermind the other aspect the pat advice doesn’t mention - managing a massive single RDMS is a goddamn nightmare. At a very large scale they are fragile, temperamental beasts. Backups, restores, upgrades all become hard. Migrations become a dark art , often taking down the db despite your best understanding. Errant queries stalling the whole server, tiny subtleties in index semantics doing the same. Yes it’s all solvable with a lot of skill, but it ain’t a free lunch that’s for sure. And tends to become a HUGE drag on innovation, as any change to the db becomes risky.

To your other point yes, replicating data “like for like” into another RDBMS can be cheap. But in my experience this domain data extraction is often taken as an opportunity to move it onto a non RDBMS data store that gives you specific advantages that match that domain, so you don’t have scaling problems again. That takes significantly longer. But yes I am perhaps unfairly including all the domain separation and “datastore flavor change” work in those numbers

[ericbarrett]

I have managed and written tooling for RDBMS from dinky GB-sized up to the multi-thousand-shard PB-scale. What you're saying is absolutely true. What a small team with vision can do when they see the ramp coming pays off 100-fold just a year or two in the future.

I think this kind of anticipation was part of Pinterest's early success, for example. They got ahead of their database scaling early and were able to focus on the product and UX.

[zie]

I think we are basically in agreement about everything, but coming from different perspectives. There is no "right" answer, but pre-mature optimization is almost always the wrong answer.

[re-thc]

And what's a more scalable solution? (in your mind)

[dalyons]

unfortunately i have no direct experience with anything that i would consider a direct replacement for the generic utility of postgres(or any RDBMS). Mostly I have been involved in moving specific domains to storage technology that has opinions that work well with the problem at hand.

eg if it looks key-value ish, or key + timestamp (eg user transaction table), dynamodb is incredible. Scales forever, never have to think about operations. But not generally queryable like pg.

if it looks event-ish or log-ish, offload to a redshift/snowflake/bigtable. But append only & eventually consistent.

if you really need distributed global mutations, and are willing to pay with latency, spanner is great.

if you can cleanly tenent or shard your data and theres little-to-no cross-shard querying then vitess or some other RDBMS shard automation layer can work.

There are a few "postgres but distributed" dbs maturing now, like cockroach - i havent personally used them at a scale that i could tell you if it actually works or not though. AFAIU these systems still have tradeoffs around table layout and access patterns that you have to think about.