Hacker News new | ask | show | jobs
by GordonS 1619 days ago
50TB is big. Bigger is possible I'm sure, but I'd guess 99.something% of all PG databases are less than 50TB.

If someone here commented they had a 2PB database, I guarantee someone else here would be like "pfft, that's not big"...

3 comments

The OP message could have better said that 50TB databases are common these days when single metal or 24xl I3en or I4* instance on AWS can hold 60T raw.
it's more than big enough to cause big problems / risk days of downtime to change, yea. 50GB is not big. 50TB is at least touching big - you can do it on one physical machine if needed, but it's the sort of scale that benefits from bigger-system architecture. 50PB would be world-class big, hitting exciting new problems every time they do something.
With 50TB, and if you were doing a full text search, wouldn't the entirety of the index have to be held in memory?
No. Full-text indexes exist.
You can also do an incremental/streaming search. Lots of ways to avoid loading it all into memory at once, yeah.